Your CLAUDE.md Was Measured, and It Was Roughly Net Negative
Every tutorial in this category opens with the same advice. Write a markdown file describing your architecture, your conventions and the intent of the repository, commit it next to the code, and the agent loads it at the start of every session. Anthropic ships a command that generates one for you. It costs almost nothing and it demonstrably changes behaviour, so the advice looks obviously correct.
Then somebody measured it properly, and the results are worth reading before you spend another afternoon on yours.
What the evaluation found
The ETH Zurich work found agents were highly compliant with tool specific instructions, using a named package manager 160 times more often when the file mentioned it. That part supports the advice, and it is the part everybody quotes.
The rest of the findings go the other way. Machine generated context files lowered success rates and raised costs by roughly 20%. Human written files bought about 4% on the harder benchmark. Claude Code was the only agent tested where even developer written files failed to beat having no file at all.
The most revealing result came from a control. When the researchers stripped existing READMEs and documentation out of the repositories, context files finally started helping, by a consistent 2.7%. Which means that most of the time, the file was restating documentation the repository already had.
Four structural reasons this happens
They are suggestions rather than rules. The documentation is explicit that the file is context rather than enforcement, and analysis of the Claude Code system prompt found the content wrapped in a note saying it may or may not be relevant. Nothing in the loop fails a build when the file is contradicted.
Adherence decays inside a session. Compliance runs above 95% in the first two messages and falls to somewhere between 50% and 60% by message ten. A separate 2026 study across 4,416 trials at six conversation depths measured constraint compliance dropping from 73% at turn 5 to 33% at turn 16 with no memory mitigation. A rule followed a third of the time is not a rule.
They consume the budget they were meant to save. These files load at the start of every session and sit in the window alongside the conversation. Files past 200 lines both consume more context and reduce adherence, which is the wrong direction on both axes at once. Anthropic’s own checkup now proposes trimming exactly the content a model could derive from the codebase itself, meaning directory layouts, dependency lists and architecture overviews.
They go stale with no failing build to announce it. An engineer hand writing what a folder is for means the person did the understanding and the model did the typing. Nothing checks the file against the code, so it drifts one commit at a time, and the version somebody wrote in March is quietly wrong by June while still being loaded into every session.
There is a fifth problem that only appears at organisation scale. These files have to be maintained on every commit in every repository. Forty developers across a hundred repositories is not a documentation task anybody completes.
The uncomfortable finding about prose
There is a piece of retrieval research that cuts against the instinct behind a tidy architecture file.
Chroma compared two versions of the same haystack, one preserving the natural flow of ideas and one where sentences were randomly reordered to hold the topic without logical continuity. The expectation was that coherence would win. The result went the other way, with structural coherence consistently hurting performance and removing local coherence consistently improving it, holding across all 18 models tested.
The explanation with the most support behind it is about distractors. When the needle is semantically distinct from its surroundings, models retrieve it reliably, and when distractors are semantically similar to the answer, accuracy drops sharply, with the drop widening as input grows.
Apply that to a well written architecture document. It is coherent prose about your codebase, which makes it a distractor factory by construction, since every paragraph in it is a plausible near match for any question about that codebase. A graph slice returning a dozen facts with a file and a line attached carries almost no narrative coherence, which puts it nearer the condition that measured better.
The effect was described as slight though consistent, and the experiment ran on essays and academic papers rather than on source code or architecture documents. Nobody has replicated it on either. It undercuts the instinct rather than proving the case, and it is better to say that than to overclaim it.
What replaces the hand written file
ByteBell is a verification layer generated on top of a codebase of any size below 10 million files, and it exists because the derived version of a context file is the only one that stays true.
Cheap open source models read every file once at indexing time. An agent detects the boundaries of each language element, stores them separately, then analyses each one in the context of its file, its module, its repository and across every repository. For each file the layer records what it does, why it exists in business terms, what users lose if it fails, the capabilities it provides, its edge cases, the contracts it provides, its preconditions, and the specific places where the behaviour is genuinely undecided.
Three properties follow from deriving rather than writing.
It cannot go stale, because it re-derives on every commit and only what changed gets recomputed. There is no version somebody wrote in March still being loaded in June.
It is checkable, because every claim carries the file and the line it came from. A statement you doubt takes about 10 seconds to confirm rather than an act of trust.
It is retrieved rather than resident. The layer is served over MCP, so an agent pulls the slice it needs at the moment it needs it instead of carrying an architecture overview in context for an entire session. That removes the budget problem instead of adding to it.
And then it does the thing a context file never could
A markdown file describes your system and never checks it.
Because the layer holds what each file promised, a change can be compared against that promise. We trace what a change touches, generate tests for it, run them, then review each modified file against its indexed base rather than against what the diff claims it changed. Collateral damage is checked across the other repositories reading the same helper before anyone approves. One full pass came to 87,001 tokens at $0.11.
Indexing runs about 0.89, against roughly $30 by brute force. Accuracy of code holds above 93% while token spend drops by roughly 80%.
What a context file is still good for
The 160 times result is real and it points at the right use. Keep the file short and keep it to the things a model cannot derive from the code. Which package manager, which branch to target, which service is off limits, what your team decided not to do and why. That is genuine intent, it is not recoverable by reading the repository, and it belongs in version control.
Delete the directory layout, the dependency list and the architecture overview. Those are the parts the model can derive, and the parts most likely to be wrong by the time somebody reads them.
The honest limitation
A derived layer is an inference about what code means rather than a statement from the person who wrote it. Where the code is genuinely ambiguous we report the ambiguity rather than resolve it, and there are cases where a human writing one sentence of intent would simply have been clearer. Every claim carries a file and a line for that reason, and there is a window after a push where a claim can lag the branch.
We also do not cover every language. Mojo is not indexed today.
This is what ByteBell builds
Something to try this week. Open your CLAUDE.md and check three specific claims against the current code. If any of them is wrong, it has been wrong for some number of commits, it loaded into every session during that period, and no build failed to tell you.
ByteBell is the verifiable context layer for code. We run the LLM compiler pattern, a one time pass where a model reads every file and lowers it into a verifiable code IR capturing purpose, business context and cross repository relationships, on your own infrastructure through Docker, with your source never leaving your environment. Every engineer, on any copilot, queries the same representation through a single MCP url, and every agent edit is checked against it before it lands, using per file SHA-256 diffing so only what actually changed gets examined again. Across 46 Kubernetes ecosystem repositories and 150,000 files we measured roughly 10% higher accuracy at 70% lower cost, on about a fifth of the tokens.