Why code still can't tell you what it's for

For a while I believed I could understand code without reading it, so I built a tool for that as part of AIND, the method this blog is about, and spent about four weeks of intense coding and testing on it. When I measured how much good it did for coding agents, the answer was: no measurable benefit. I see two reasons. One I observed in a detailed analysis of the agents’ sessions, namely that agents do not read code the way people do. The other is my interpretation: the generated statements described what the code does, but not why, because intent does not live in the code. That is why design has to come first, separate from the coding.

Why I tried

To understand code means to understand the intent behind it. Intent does not describe what is there; it is a set of rules and decisions, motivated by requirements, that together set a target structure for the code: its rough shape and, indirectly, the place where each extension belongs. These rules also cover what does not exist yet, because they say where the next feature goes before anyone has written it. Code could in principle match that structure completely, but in practice it almost never does, so the code is nearly always an incomplete image of the target.

Without these rules you cannot develop the code sensibly. Only someone who knows them knows where the next change belongs and can make it fit what exists, without side effects nobody wanted.

When a coding agent writes large amounts of code that nobody reads, you need another way to understand it, and the obvious idea was to let an agent analyse the code and derive the rules and concepts from it. I called the result claims: statements of what a part of the code does and how it interacts with other parts, each pointing to a specific place in the code and written by an LLM that had read that code completely. In AIND, the claims were meant to answer the first question of this blog, how to give up control over the source code without losing track of it. Together they came to about an eighth of the code’s volume, counted in characters, and could be read selectively.

I assumed they would help the agent too: knowledge produced once, kept current and used in every session would let it read less and decide better. The code, about 135,000 lines across a language server, an editor and three tools, not counting tests, had grown task by task. Each task came with its own analysis and design, but those designs stayed scattered across many documents and never became one approved, maintained specification. So the claims were reconstructing something that never existed as a coherent design.

What I measured

I gave coding agents a set of tasks and let them read the code in three ways: by searching and reading the files only, with an additional map of the code’s structure built by a parser, and with the claims on top of that. An independent review then counted the defects in each result.

I ran two studies, one with tasks that named the target and one with tasks that named no file or class. In neither did the claims reduce the defects; they only cost more: in the first study 19 percent more tokens and 34 percent more calls than plain reading, in the second 31 percent more tokens than the parser map alone. The scope was limited: one codebase, the language server, about 50,000 lines, one model, five to eight tasks per study and one run per combination.

In the first study every first result contained defects, whichever way the agent had read the code. The extra context did not prevent them; each time the review found them.

The claims themselves were unreliable. Of 133 claims in four modules, 13 contradicted the code, about one in ten, and another 21 were true but not supported by the citations they gave, so about one in four could not be relied on. Producing one claim and checking it once cost around 52,000 tokens. Whether claims help human readers I did not measure; my impression is that they often help with orientation, but that is a gut feeling.

Agents read differently

A person reading code typically forms a hypothesis about what the system does and then verifies it, so the picture comes first. The session analysis showed something else for the agents: they searched and read in short, targeted steps, fast and surprisingly precise, without building up an overview first, and gathered only the evidence for the question in front of them. A claim is written for a reader who builds a picture, for the agents the claims added reading without changing the result. They just located the code files and read them, the additional explanations in the claims obviously did not help them in solving the task.

What the code cannot say

From here on, this is interpretation, not measurement.

The claims described what the code does, in detail. They did not describe the rules for what is not there yet, and a description derived from the code cannot. Naur argued in 1985 that the theory of a program lives in the people who built it, not in its text. Eden and Kazman (2003) describe the difference: architecture states rules that hold across the whole system, including parts that do not exist yet, while the implementation is only what exists today. Many different rules fit the same code, so you cannot reconstruct them mechanically. Reconstructing them is a creative act, and the result has to be managed, updated and verified against the code.

Keeping claims correct is also expensive. A claim can be trusted only after at least one round of review, and another after each correction; in code that keeps changing, that never ends. So I took claims out of AIND’s coding path. What stays is the parser-built map. It costs no model tokens, and with it an agent in the second study reached a named place in four to five calls, where guessing the names took fourteen.

An agent finds by searching where things are and what they do, but not where something new belongs, because that is not there to be found. So it puts the new thing where it costs the least effort right now.

Reverse the direction

The previous post argued that the link between intent and code has to be written down before or while the code is created. The claims tried the opposite: they tried to recover it afterwards, and it failed.

The answer is to separate design from coding: the design comes first, binds the code and follows rules that do not depend on any concrete coding. Murphy, Notkin and Sullivan proposed this for architecture in 1995: the engineer states a model, and a tool reports where the code agrees, differs or is missing. A rule for where the next thing goes has to exist before the next thing is written, so there is no alternative.

This is not what current practice suggests. Vibe coding gives the agent a goal and lets it run. Most spec-driven approaches specify one feature at a time. Where they have system-wide documents, such as constitutions, steering files or AGENTS.md, these hold conventions and principles, not a structure the code is checked against. Build a system feature by feature, and you get the decay described in the previous post: each change lands where it is cheapest, and the structure of the whole is whatever is left behind. The design does not help the agent by being read, the way a claim was meant to; it helps because the code is checked against it.

AIND sets the scope at the whole system. In the previous post I called the units of design Components and the units of code Modules; the Components come first, and the Modules are checked against them. That is what “specification” means when I say AIND aims to reduce the human role to specification and verification. Missing that was the biggest weakness of my approach so far, and the Components are meant to fix it. How that works in practice is the subject of the next posts.

This is uncomfortable, because the design becomes a piece of work of its own, done before the code and kept after it. The claims promised to spare me exactly that. Experience also says it seldom works: in most software projects the initial design is overtaken by the code fairly soon, and from then on the truth lives only in the code. Design rarely keeps the leading role I give it here.

It can work this time, for three reasons. First, the code is generated by machines and checked against the design, so deviations are caught instead of piling up unnoticed, as they do when people change code by hand. Second, the design has a value of its own. People could do without it because they filled in the missing design themselves, if imperfectly: software maintained for years by changing teams often shows the same symptoms as the decay in the previous post. An agent does not fill in the missing design; it only picks the cheapest place, so without a design the code cannot be developed further efficiently. Third, keeping design and code in line gets cheaper: checking code against a stated design is a narrower task than reconstructing one, much of it is mechanical, and AI can help with the rest.

The hype says agents will take over design as well as code. I am not sure. Two questions are still open: whether hallucination can be reliably controlled, and whether an LLM is capable of real abstract thinking. The one claim in ten that was wrong shows that the first is not answered yet, and today’s LLMs have no satisfying answer to the second; whether that lies in how they are built or only in their current implementation, I can’t judge. Until then, software design remains a task for humans.

What is not settled

Some decisions are in the code. A model can sometimes recover explicit decisions that sit close to the code. I do not claim otherwise. What it cannot recover is the big picture: the general rules for what does not exist yet.

The reversal itself. I do not know whether stating the intent first stays cheaper than the code it governs. Nothing here measures it.

Frozen code. Claims may save effort for code that no longer changes and is only reverse-engineered. The cost of generating and reviewing them is paid once. Claims can also serve as hypotheses that a person confirms or discards while reconstructing the structure. I have not measured whether the review rounds converge, or how many errors remain after them.

Sources

1. Naur, P. (1985). Programming as Theory Building. Microprocessing and Microprogramming 15(5), 253-261.
2. Eden, A. H., Kazman, R. (2003). Architecture, Design, Implementation. ICSE ’03, 149-159. https://dl.acm.org/doi/10.5555/776816.776835
3. Murphy, G. C., Notkin, D., Sullivan, K. (1995). Software Reflexion Models: Bridging the Gap Between Source and High-Level Models. ACM SIGSOFT FSE. https://dl.acm.org/doi/10.1145/222124.222136