I have largely avoided discussion of LLMs and Generative AI on this blog. But of course, it is impossible to escape thinking about them. Moreover, my day job is as a programmer building AI agents. To be clear, I’m not in one of “the labs,” I don’t have any special insight into how the models are made. I do have some sense of the consumer side of things, what it’s like to use the (publicly available) models in software products.
Rapid-fire Takes
I assume you’re familiar with LLMs. Text goes in. Funny things happen with numbers (weights). Text comes out. The weights happen when a mommy training corpora loves a daddy reinforcement learning process very much. As training corpora have come up against limits of the amount of data that exists on Earth, and as moar training and moar weights started to display sharply diminishing returns, the last couple years of model improvements has come more from “post-training,” usually in the form of giving the program massive generated test suites in say programming or mathematics and tuning the weights until it aces them, and “test-time compute,” letting the model babble to itself internally before producing a response.
Before we get onto topics where I actually have, I think, something interesting to say, let’s do a quick round-up of takes.
How smart are these models, really? I think it’s hard to compare them directly to humans. Clearly on some, particularly math and programming, the frontier models have superhuman cognitive performance. Other things (writing poems, say), they are definitely worse than a skilled human. At least in programming, they are not yet a complete substitute for human labor, but replace enough of it to completely change the character of that work.
Will models self-improve into being super geniuses that do obsolete all human cognitive labor (and physical labors, once robotics catches up)? Maybe.
Will that lead to human mass extinction? Maybe.
Will that lead to a post-scarcity utopia? Maybe.
Even if neither happens, will we experience a global economic collapse as human labor gets out competed? Maybe.
Will we instead see an unprecedented economic boon from this technology? Maybe.
Will the massive capital outlays on the AI buildout see enough returns on short-to-medium terms that investors care about (and which matter given hardware lifecycles) to avoid a railroad mania / dot-com bubble style crash? Maybe.
Are these models conscious now? I would be surprised, but maybe. Will future iterations be conscious? Maybe.
What does that mean in terms of our moral relations to these potential new minds? No clue.
What should we be doing to improve our chances that this technology turns out well rather than killing us all or destroying our economy? No clue; some type of slowdown seems sensible.
Isn’t AI “art” a moral affront to the spirit of humanity? Isn’t the technological development built on a massive foundation of theft and a further, horrid example of capitalist enclosure of the commons and an all out war of the owner-class against the laborers whose living, needing human flesh they hope to replace with an army of mechanized, perfectly compliant minds? Does that not make now our best and perhaps final chance to usher in the revolution? Idk.
With that enlightening romp through the discourse completed, we can start saying more useful things.
Agents, a Brief Introduction
At their core, these models are next-token generators. However, the industry has standardized around a “messages” format for interaction. You can think of this as roughly meaning that the text the model is asked to complete takes the following format.
Here is some context and rules for you to follow in responding to the user: … (The system prompt.)
Now, the user’s request is: … (The user prompt.)
Your response is:
What the model spits out in response to this prompt is called its completion. You get the chat experience by allowing this game to be repeated: the model produces its completion, then the user adds a followup question and the “assistant” answers. Now, the model has no memory so “answering followups” works by feeding the model the transcript including its past responses. (In order to make long back-and-forths feasible, model providers will generally cache the intermediary responses for a period.)
As the chat grows longer, the computational work, the amount of memory, grows along with it. (Indeed, the work grows according to the square of the input length.) So longer chats are more expensive and past a certain length become prohibitive. Further, response quality degrades as the input grows larger. The model only has so much attention so the more it’s asked to consider, the more likely it is to fail to consider part of the prompts. The amount of text a model can take in its prompt and still produce output is its “context window.” The point at which the input is too long and output quality is noticeably degraded is sometimes called the zone of dumb. Context windows will vary by model and provider, but frontier models now routinely offer context windows of a million tokens (roughly 3/4 of a million words, about 10 novels long); the “zone of dumb” is an informal measure, but it starts roughly 30-40% of the context window. A context window of a million tokens is absolutely enormous, but we’ll soon see that it can be easily reached.
So far we have a system that can’t do anything other than hold a conversation. In order to use the model to do things, we need to get the model to well, do things. But the model itself is just tokens-in, tokens-out, nothing else. The way around this is, in your calling program, to say something like, “if you want to perform a web search, respond with <web_search>…</web_search> and I’ll look that up and give you back the results.” And then to program your calling function to look for text in the completion that takes that form and paste the query into Google and then to put the output of that query in the next prompt back to the model. Here web_search would be an example of a tool, the controlling program that prompts the model and pastes back in the tool output is a “harness.” The harness + model + system prompt + tools is an agent.
Tool calling is the basic functionality that a harness provides. Nowadays, there are many more standard features. Compaction, handling the limits on context window by calling a model to summarize the transcript when it starts getting too big. Subagents, letting one of the tools the model can pick from be one that spawns a new agent with a prompt and responds with its final output. This can get you more out of the context window (as it keeps noise out of the parent agent), delegate problems to different models (easy problems to cheap models, hard ones to expensive ones) and avoid the “zone of dumb” as subagents can spawn with fresh contexts. If you’ve heard of “agent swarms” this is typically a related but not identical capability where the harness gives the model the ability to send messages to other agents.
Harnesses and models have co-evolved. In the bygone era of 2022, 2023, even the best models struggled to reliably produce the kind of structured output needed for tool-calling to work. By 2024, you had reliable tool-calling, but context windows were small, models weren’t that smart yet, so you would hit the zone of dumb pretty quickly. But by 2025 you had very powerful models with big context windows that were trained to tackle long, multistep agentic work. So by 2026 you start seeing things like coding agents successfully porting large projects from one programming language to another, or decompiling videogames, or math agents solving millenium prize problems.
Why Isn’t Everything Run By LLMs Yet?
If models + harness can achieve such impressive results, why have they not completely taken over the economy? To put a finer point on it: I’m a programmer, which is the domain that these models are best at; why do I have a job? The skeptic’s answer is that the models aren’t actually good at things, it’s just an illusion we’re tricked by “stochastic parrots” that produce fluent and confident output. I think “the LLM believers have given themselves AI psychosis by flooding their brain with slop” was a kind of plausible thing to be saying a few years ago, but now the models basically are that good. It’s a line that only sounds plausible today if you’ve only touched the free version of ChatGPT or Gemini. Still, if models are that good, why are there still humans typing on keyboards. Even the AI labs and most “AI-forward” firms still have a bunch of human employees.
The short story is that an agent’s job is impossibly demanding. You can think of the model as a temp, remote worker receiving instructions to perform a specific job via email. They have their instructions, but they are otherwise behind a veil. The model’s training has given it some context, but only that which is publicly available and written down. The context window means the prompter can’t just stuff everything the model might possibly need to know for the task at hand. So you either need (a) to very, very carefully scope the problem to one where you can prepare just the right context for the task ahead of time or (b) give the agent tools to search for relevant information on demand or (c) force a human to supervise the agent and feed it extra context as needed. All of these are tricky and expensive to implement.
The disadvantage an agent has over a human, even if the model outguns humans (in some tasks) cognitively is that humans have memory and like talking to each other. So while a human might start out without the knowledge required to fully do their job, they can ask a coworker and then remember that for future tasks.
You can approximate this, kinda, for an agent, e.g. by giving it the ability to take notes (or automatically generating notes from transcripts) that it can search through in future. The challenge is (a) the veil of ignorance that makes it hard for the agent to complete an ambiguous task also makes it hard for it to take genuinely helpful and accurate notes, (b) those notes easily go stale, or (c) become a complete mess of amendments and cross links (“decoherence” is the keyword here) so that the agent starts struggling to understand its own notes to itself. (Mood.)
This is not to say that it is impossible, by any means, to get an agent to do useful work. But it is to say that there is a high implementation cost and even a substantial maintenance cost. And even once you have an agent that can do real work, you’re not out of the woods yet. To run through the remaining difficulties in a quick list.
The curse of automation. A general problem with technological automation is that it introduces more centralized failure points. If you automate logistics between factories, failure of that automation can harm the whole network whereas before automations failures were more likely to be localized. In the same way, a failure mode in an agent that has automated a task at a firm may damage the firm’s work as a whole, rather than the limited damage that a single bad worker could do.
Security. It is well known that models suffer from prompt injection. As I mentioned before, the separation between different part of the prompts (system prompt, user, assistant messages, tool outputs) are conventions enforced by text. Nothing enforces that the model only follows instructions given by the system or user; indeed you wouldn’t want to enforce this in general because “what the model should do” is often something it has to look up, i.e. will be in the tool output. This fact makes it possible for an attacker to slip in their own instructions somewhere a model may fetch when working through a task. And if you’ve (intentionally or unintentionally) given the agent the ability to leave itself notes (or to do so in effect by modifying its environment), a bad prompt injection can have the agent copy it, poisoning future sessions.
Security (pt 2). There’s this whole “alignment problem:” models don’t always do what you want, whether this is for true misalignment (maliciousness), ignorance (the model makes a bad guess in the face of ambiguity), or stupidity. Whatever the case, even in absence of hostile context, the agent will occasionally misbehave.
Security (pt 3). You need a model for an agent. But the normal way to get access to a model is to use one some cloud provider is hosting. But this means sending often sensitive data over the internet. The alternative is to self-host an open weight model, but this very, very, very, very expensive infrastructure to host a model that will be noticeably worse than the ones you can buy.
Security (pt 4). Agents are also just software and so inherit all of the normal difficulties of software security.
Automation bias. A sensible firm will run an agent in shadow mode, having the agent do the same task as a human but let it only make a recommendation rather than acting; then you compare what it would do against the human to see if it makes sense. Often, firms will quickly skip to a Copilot setup: have the agent work alongside a human and make recommendatios for that human to implement or approve. Even before agents, we knew humans have an “automation bias:” we trust automated recommendations somewhat blindly. However, this bias gets worse for the confident, fluent text that agents tend to produce. So even before the agent is acting autonomously, it can do damage to the firm’s operation.
Cost. While the “cost of intelligence” has been coming down at an impressive clip, using especially the best models will incinerate cash. So unless the task is so valuable that cost is no object (and the demise of tokenmaxxing suggests that there are very few such tasks in our economy), you’ll want to spend time optimizing model selection and context usage as well as adding cost monitoring and controls to make sure you’re actually seeing a return on your agent.
“Slop factories.” The fact that agents are horizontally scalable and, if you want, always on means you can get a massive quantity of output. But a massive quantity of low-quality output can, in many ways, be worse than no output.
“Reverse centaurs.” Cory Doctorow notes that agents (like previous automation, only more so) can transform work into a degrading “automation minder” where the worker spends their time doing the stressful work of making sure a bunch of agents aren’t fucking up the jobs they use to do themselves. In the worst case, workers describe their new job as “pressing enter” on approval prompts. For the purposes of this list, I’m concerned with the business costs not the moral costs of agentic transformation. But obviously making your workers’ jobs miserable tanks morale. To the extent that they had knowledge, skills, creativity that your agent lacks (hint, creativity and contextual wisdom are things even the best models are comparatively bad at), them disengaging or leaving (or you getting all eager beaver and firing them) is obviously bad for business.
Corporate bureaucracy. The internal processes of a modern corporation look like a bewildering array of systems poorly wired together and split across organizational fiefdoms with byzantine office politics. Not everything is even digitized. Making agents that can work in such environments is as much a matter of transforming organizations. I have worked on a few projects where it has taken longer for IT to grant an agent access to an internal system than it took to develop the agent in the first place.
Software is hard. Agent building is 80% traditional software engineering. Software engineering is and remains hard, despite how good coding agents are.
Teaser
I had intended to illustrate this discussion with an extended case study on coding agents, but this post is already at risk of crossing over into medium effort territory, so I think I’ll save that for tomorrow.