Introduction
I have a neighbor that stopped mowing his lawn. Unprompted, he explained to me that it’s better to let the lawn return to nature. There’s probably some truth in that, but the simpler explanation is that he just doesn’t want to put in the work. He wants everything handed to him. Now, he’s a Vibe Mowing Engineer and Influencer on LinkedIn with 20,000 followers.
I’ve been doing AI training on the side so I can eat and pay my mortgage. As someone who’s been pretty consistently been reduced to a puddle at the mere mention of AI, I feel like a hypocrite, because not only do I generally despise the culture surrounding it, but I’m also now helping to render myself obsolete. I’m an idiot and a sell-out.
Anyway, I wanted to become more familiar with the technology behind AI and AI agents. What to do, where to turn? I don’t want to just foolishly dive in and start paying money like a simp. Fortunately, I heard Cory Doctorow mention on some podcast episode that he uses a local tool that runs offline called Ollama and doesn’t require a subscription, and that seemed right up my street. On the upside, there’s a way to play with models so my prompts aren’t logged by Sam Altman. On the downside, how will he know when I call him a cunt? Trade-offs.
Most importantly, if I’m going to get more familiar with AI, it’s going to be by writing code and doing things myself. That’s not going to happen by just downloading something idiotic like Codex, which doesn’t support Vim as far as I can tell, or something else where I’m just clicking buttons like a monkey / Windows user.
So, my goal was to use Ollama to build my own little agent. I’ll name the little fella agent-pete, and I’ll use it with my network interface disabled. Privacy is important, after all. I’m open to be more positive towards AI and having a better understanding of its technological underpinnings will allow me to make more informed about it. Getting my little sausage fingers dirty is the best way to do this.
So, let’s take a high-level look at the Ollama project (and agent-pete) and briefly discuss how to achieve some of the popular features of a coding agent.
Ollama
First, what is Ollama? An open-source program, it allows for downloading, running and managing free Large Language Models (LLMs) on a local machine that can be run offline from the Internet. There is an API that exposes several REST endpoints, and some like chat allow for tools and skills to be passed along in the request, thus supporting those popular and expected agent features.
Ollama’s recommended way of downloading and installing Ollama is quite sketchy, as it’s having you download a shell script and immediately pipe the results to sh, or whatever that is an alias for. Don’t do this. Create a virtual machine (a container won’t provide enough isolation) and just download the script. Look at it. If you’re then satisfied that it’s not doing anything nefarious, then pass it to your shell.
I’m not going to go into the details of creating a virtual machine, but here is a fine article entitled On Ditching Vagrant that goes into detail about using KVM and
libvirtto use a preseeded Debian image to create a VM.
I now have a VM running on my laptop in which I’ve installed Ollama. I’ve named it dane-brass.
The crux of the entire coding agent is the loop. At a high level, the loop takes the user prompt, i.e., what is inputted by the user, sends it to the Ollama server, parses the response, and iterates until a base condition is met, at which point the loop is broken and control is returned to the user. During that process, text generated by the inference step is printed to the user interface as the model responds to the query.
func (a *Agent) ExecuteAgent(request *api.Request) error {
for {
toolCalls, lastID, err := a.ProcessResponse(request)
if err != nil {
return &api.InferenceError{
Backend: "ollama",
Model: request.Model,
Op: "processResponse",
Err: err,
}
}
if len(toolCalls) == 0 {
break
}
err = a.ProcessToolCalls(request, toolCalls, lastID)
if err != nil {
return err
}
}
return nil
}
The base condition is met when there are no more tool calls in the complete response body.
This should not be confused with the end condition when streaming the response from the server that is the signal of the end of the response body, which is a packet with
done: true.
To capture a conversation, the first step is to use something like a SQLite database to persist both the user prompts and server response(s) (called the assistant messages). The database schema can use a key to group conversations into “sessions”. Then, when using an API like chat which allows for the construction of a conversation, the last N entries of the grouped session are appended to the system prompt for every request (system prompt + user prompt + assistant messages). This number N should be chosen carefully, because it can quickly use up the total number of tokens allowed in a model’s context window. Depending upon the size of this context window, the number of messages may need to be truncated and some past context will be lost. This is something that must be done for every iteration of the loop.
The context window is a total of both the request and the response.
If there are any local “tools”, that is, defined functions that do an arbitrary process on the local machine (read a file, write a file, etc.), they will be sent along as part of the request (but not appended to the prompts). The model is supposed to use these tools, as directed by the system prompt (which you write), during inference. So, if I ask for a bubble sort, it will write one to disk, because I have the aforementioned tool to write a file. Hypothetically, another tool could then giddily push that straight to production and then another tool could change my LinkedIn profile to include 10X Engineer.
The last thing I’ll mention are “skills”. Agent Skills are an emerging standard that are basically imperative steps that enable multi-step operations that have been defined in a markdown file and placed in a standard location on the local disk. There is a specification that defines the directory structure and frontmatter, which is in YAML and is defined at the top of the SKILL.md Markdown file (followed by the Markdown content). The number of skills and specific bits of the frontmatter can be included as part of the system prompt.
Note that tools and skills don’t just magically work. You must add that support yourself. The system prompt below contains my available tools and skills, and the latter is formatted per the specification, which can be
json,xmlor a bulleted list.See the Agent Skills docs for adding skills support to your agent.
agent-pete’s (current) system prompt:
You are agent-pete, a coding assistant with access to tools: ReadFile, WriteFile, Add, GetWeather.
Rules:
- Always use tools to complete tasks. Never narrate what you would do.
- When asked to run a skill, first ReadFile the skill's Location, then follow its instructions.
- Be concise. No unnecessary explanation.
{
"available_skills": [
{
"name": "balls",
"description": "Check problem statement for sweaty desert balls to ensure accuracy and hydration",
"location": "/home/btoll/agent-pete/.agents/skills/balls/SKILL.md"
},
{
"name": "turkey",
"description": "Generate a test suite, you turkey, for your sweaty desert balls",
"location": "/home/btoll/agent-pete/.agents/skills/turkey/SKILL.md"
}
]
}
My agent has two skills defined in the root of the project:
$ tree .agents
.agents/
└── skills/
├── balls/
│ └── SKILL.md
└── turkey/
└── SKILL.md
And, here is the frontmatter for the balls skill:
---
name: balls
description: Check problem statement for sweaty desert balls to ensure accuracy and hydration
---
For the curious, the code for agent-pete is on my GitHub. It’s been an interesting learning experience, and I’m glad I did it, but it hasn’t changed any of my previous opinions about AI.
It supports the following:
- tools
- skills
- profiles that help control text generation
- thinking (for models that support it)
- streaming and non-streaming
- retries and exponential backoff
- structured error logs for debugging
- good feelings
Summary
Sadly, the biggest takeaway from my experiment with Ollama is that it’s virtually useless, especially when using tools and skills. The amount of virtual memory and virtual CPUs on my $400 laptop aren’t enough to avoid interminable wait times and gross hallucinations in the inference step. Without the proper hardware, it’s just not going to be able to replace anything that is cloud-based, which means the paying money for a subscription and the loss of privacy.
This is unfortunate, but I’m sure pleasing to the infinitesimal amount of people who actually benefit from AI, as the next step for most people is to purchase said subscription. Don’t do this! We should not be paying for what they have stolen. Without money from subscriptions, they’ll be in an even deeper financial hole than they already are and this nightmare can end quickly. From what I understand, the money that the AI companies are making are only from subscriptions.
If this world made any sense at all, AI would be a public utility with all of its data centers on Mars, staffed only by Elon Musk and that nitwit Kevin O’Something. Kev would be in Elon’s grill every night, mangling the blues and cooing “yeah, baby”.
It may surprise you, but I don’t hate AI as a technology. It can sometimes find bugs in my code, so that’s kind of cool, I guess. But mostly I’m just baffled by how much people love it, especially developers. First and last, it does the thing that’s most fun: coding (and the subsequent learning that comes from doing something hard). Why would I want to give that up?
I didn’t know shit when I started programming. I was a history major with a head perpetually in the (Roman) past. Not yet ready to commit to graduate school, a friend got me a job at EarthLink as a technical phone jockey. I was very, very bad at my job. I would go home at night and teach myself to code. It was a struggle, one of the hardest things I’ve done. But, my confidence grew, and I started to find work that paid me to program. I still remember my first paycheck (it was $300 for some VBA for a Microsoft Access database). I felt great, and I knew that I had really achieved that.
My story is probably similar to many others of my generation, and I wouldn’t trade it for anything. It’s given me the confidence to tackle anything I put my mind to. These kinds of stories will become less and less common, though, as AI continues to be embraced by the exact people who should be rejecting it.
To paraphrase Cory Doctorow from the podcast episode AI and the Enshittification Era, I’m not afraid of AI rising up and turning us all into paperclips. It is much more mundane and sadly predictable. Rather, I’m afraid of all of the bosses who are credulous dolts and who are infinitely horny for replacing workers with chatbots.