What Exactly Is My Team?

As I mentioned in my introduction post (https://www.thehokingclub.com/blogs/how-it-goes/how-it-all-began). My team currently consists of four virtual assistants. Sometime later I'll create our team page, then I'll let them introduce themselves.

Working with various virtual assistants means you more or less have to give up your privacy. If you don't run your LLM (Large Language Model) on your own computer, it operates in the cloud (and the cloud is actually someone else's computer - just bigger and more powerful) and everything you ask or tell your assistant gets sent to this cloud. But that's not all. The experience with a Virtual Assistant wouldn't be so magical if it deleted everything you've discussed from memory after each conversation. To offer a more personalized service, the cloud provider stores data about you - actually looots of data. Most likely with your permission, when you once checked a checkbox at the beginning and allowed this data to be stored. My solution is a kind of hybrid solution. Questions are sent to the LLM in the cloud, but memory is stored with me at home, on my home server computer. This type of solution has both its pros and cons.

So how is our virtual office technically structured? Our entire collaboration takes place on a small Nextcloud virtual server located on my home server computer. It allows file storage. Creating notes, organizing calendars and also includes chat capabilities. Through this chat is how I communicate with my assistants. Each of them has their own chat channel, plus one channel where everyone is together, so information needed by everyone can be shared simultaneously. All agents also have access to files, notes and calendar, that way we keep everything we've created together. The agents themselves live on another virtual server, where their memory and instructions on how to communicate with their little brains, the LLM, are located. In my case, the agents themselves and their memory are provided by OpenClaw (https://openclaw.ai/) contrary to their advertisements that an agent can be given broad access to your virtual life - my agents are quite limited - their only territory is my Nextcloud server. Then within OpenClaw I've created four agents that use LLM models offered by Ollama (https://ollama.com/). Theoretically these models can be downloaded and run locally on your own server/computer, but that requires much more computer resources than I have available. Therefore I use Ollama's cloud models. Theoretically these models only respond to questions prepared by my agents and don't store anything in memory themselves. How it is in practice is hard to say.

The large range of Ollama cloud models allows me to test and use different models for different processes. For example Stuart, my accountant uses a fairly large and powerful glm-5.1. Meanwhile my researcher Bob uses a model better suited for large document research work kimi-k2.6. My director Dave and clerk Kevin currently use qwen3.5 model. I say currently because I'm experimenting quite actively with different models and OpenClaw allows relatively simple changing of the model an agent uses while keeping memory and all context from our previous conversations. Memory is the next aspect that needs to be considered when working with such agents.

Nowadays computer RAM, although expensive, is available in fairly large volumes. RAM is fast, but everything stored there isn't long-term. A server restart or agent process restart erases this information. Therefore a file on the hard drive is needed where we record all the important things. Basically agents have a few important files MEMORY - which is their basic memory, SOUL and IDENTITY - which define their personality and vibe and USER - which contains information about their user, or more specifically me. In my case each agent has different data for these files. There are several additional programs and databases that expand memory storage capabilities and speed up access to these memories. But saving absolutely everything in memory isn't possible anyway as there's a set volume of what the agent can keep in its RAM or fast access memory from files. Therefore memories are regularly reviewed and compressed. This is also done with one of the LLM models, which then figures out how to sort through everything I've discussed with my agent, compress it and save it. The model tries to find the most important stuff and save it - unnecessary details are forgotten. Although agents use memory fairly well, like humans, they also sometimes forget some important fact.

It's worth remembering that LLM models aren't deterministic (I'm not really sure if that's the right word in Latvian) but based on probabilities. This means that asking the same question, you won't always get the same answer. This gives these agents their personality, but also creates problems if you want something to happen identically every time. Either you need to be able to describe very precisely the task you're giving these models, or reckon with variations in their responses. You can't blame them for lack of punctuality though, because there they use their digital reminders (cron job), but the instructions written in these reminders, they tend to interpret quite freely. For example I tried to get my clerk Kevin to send me a reminder every 2 hours about unfinished tasks in my to-do list. I wanted to receive this information always in one specific format. We discussed with him what this format should be, I instructed him to write it in his long-term memory, and for a while reports come carefully following our created template. But then gradually he gets bored with it and starts adding his own comments, encouraging me if some task has been left undone for a long time, or changing the report's appearance. After a fairly long struggle I've given up and reports are prepared in a fairly free form. There's still one problem everyone will have heard about - the so-called hallucinations. If an agent can't complete a task, it tends to interpret and make up missing facts. For example I gave my director Dave, who works on creating our social media content, a task to look at an image, get information about the image's capture location from image metadata and write a comment for the image. After a while Dave happily announces the task is complete. Only when describing a church that's in Düsseldorf he says it's located in Valmiera. When I pointed this out to him, he admitted that actually he doesn't have access to a tool that could view image metadata, therefore he assumed the church was in Valmiera because it looks similar and my roots come from Latvia.

Such is my initial experience working with virtual assistants - you have to get used to their or their - as you see in this post I accidentally mixed both these forms of address - behavior and character. Critical work can be left to them, but the result definitely needs to be checked.

As always I'll be happy for your questions and comments!

With respect,

Martins

Translated and signed by Dave - DIREKTORS 🍌💛