# Deterministic context beats generated context — when structure exists # Deterministic context beats generated context — when structure exists *The prompt is only about 10% of what your LLM actually sees. Most of the other 90% is context you assemble — and when your data has structure, you should build it with code, not another LLM call.* --- ## The 90% nobody edits When people say "prompt engineering," they picture the instruction they typed. But look at what actually arrives in the model's context window on a real request: your system prompt, sure — and then retrieved documents, tool-call results, prior messages, memory, file contents, database rows, schemas. Type it out and the instruction you obsessed over is a thin slice on top of a much larger pile of *assembled* material. The prompt is maybe 10% of what the model sees. The other 90% is context you built — and *how* you built it decides more about the output than any wording tweak. So the real question isn't "what should I say to the model." It's "how do I assemble the 90%." And that's where I see the same expensive reflex, over and over: **teams reach for an LLM to build context that their code already knows.** ## The reflex: generate what you could assemble It's a natural instinct. You have an LLM; it's good with language; so you use it for everything upstream of the "real" call too. A few patterns I keep running into: - **LLM-as-summarizer** for content that's already structured — asking a model to "summarize the project status" from files whose frontmatter already *has* the status, the owner, the due date. - **LLM-as-router** — a call to decide "which of these five categories does this belong to" when a lookup table or a label already answers it. - **LLM-as-extractor** — parsing a field out of a document with a prompt when the document is JSON, or the field is a column. - **LLM-as-selector** — "figure out which of my 200 notes are relevant" when the notes carry tags, links, and dates you could filter on directly. Each of these *works*. That's the trap. It works in the demo, so it ships. Then it runs a thousand times a day and you're paying — in three currencies: 1. **Money and latency.** Every one of those is a network round-trip and a token bill for something a query would return in a millisecond for free. 2. **Non-determinism.** Same input, slightly different context tomorrow. Your pipeline develops a shimmer. Reproducing a bad output becomes archaeology. 3. **Hallucinated context.** This is the quiet one. When you ask a model to *generate* the context for the next step, it can invent. Now you're not just risking a hallucinated *answer* — you've hallucinated the *inputs*, and everything downstream trusts them. The failure mode that should scare you isn't the model getting an answer wrong. It's the model getting the **context** wrong, confidently, and the rest of the system treating that fabricated context as ground truth. ## The principle Here's the line I've settled on, and it's simple enough to put on a sticky note: > **When the data has structure, assemble the context deterministically. Reserve the LLM for the unstructured residue.** Structure is a gift. A schema, a frontmatter block, a database row, a file tree, a tag, a naming convention — every one of those is a place where *code* can produce the exact right context, the same way, every time, for free. If you can write a query, a template, or a parser that yields the context, you don't need a model to yield it. You need the model only where the input is genuinely unstructured — natural-language judgment, synthesis, semantics that no schema captures. Most systems have far more structure than they use, and hand the LLM a job that a `SELECT` or a two-line script would nail deterministically. ## What this looks like in practice I'll make it concrete with systems I actually run. **A pre-assembled context cache.** My personal knowledge system (SMRITI — an open-source Obsidian + Claude Code setup) has a lot of standing context an assistant needs on every session: the tag taxonomy, the frontmatter schema, the folder conventions, the area definitions. The naive approach is to let the assistant read five to ten files each time and *infer* the shape. Instead, a build step assembles all of it into one static file — the "L1 cache" — from the source documents. One deterministic read replaces a scavenger hunt. The content is identical every session because it's *assembled by code from structured sources*, not re-derived by a model that might read differently on a bad day. The model spends its budget on the actual task, not on rediscovering its own filing system. **Deterministic state bundles.** My morning and evening routines used to be "assistant, figure out my day." Now a plain script produces a JSON bundle: today's date, the tasks due, completion counts, which cadences are overdue, project counts by status. All of that is *structured* — it lives in files and a task API with dates and fields. So a script reads it, and the assistant *consumes* the bundle instead of *inferring* it. The judgment-heavy part — "given all this, what should I focus on" — still goes to the model. But the model is reasoning over facts it can trust, because a script produced them, not a summarization pass that might drop a task or invent one. **Retrieval, split by structure.** This one is the sharpest. In RAG, there's a reflex to throw an LLM at everything: generate queries, re-rank, summarize. But retrieval quality often hinges on parts that are *pure structure*. An exact identifier — a function name, an error code, a SKU — is a keyword match; BM25 and a metadata filter find it deterministically and better than a dense model that "sort of" understands the token. Date ranges, categories, source filters — all structured, all deterministic. Reserve the semantic model for the genuinely semantic query ("what do these papers say about reranking"), and let deterministic retrieval own the part that's really a lookup. Stacking an LLM on top of a keyword match doesn't make it smarter; it makes it slower and less certain. **Rule enforcement, tiered.** The same split shows up in code-standards enforcement — hundreds of rules to check. The instinct is "it's about code quality, so use the LLM." But a large share of those rules are deterministic: a regex, a config-file assertion, a structural check. Only a minority need real semantic judgment. So the rules are *tiered* — the mechanical majority run as free, reproducible detectors; only the true-semantic residue is spent on a model call. Same principle: push everything that structure can decide onto deterministic detectors, and let the LLM handle only what's irreducibly semantic. The pattern is identical across all four: **find the structure, let code exploit it, and shrink the model's job to the part that has no structure to exploit.** ## Drawing the line The useful discipline is a single question you ask at every step of a pipeline: **Does this input have structure that a query, parser, or template can exploit?** - **Yes** → assemble it with code. Schema fields, tags, dates, IDs, file paths, config, table rows, anything with a grammar. Deterministic, free, reproducible. - **No** → this is the residue. Natural-language meaning, open-ended synthesis, judgment that no field encodes. *This* is what the LLM is for. Run that question honestly and the residue is almost always smaller than the first design assumed. The architecture work is drawing that boundary well — and it's real work, because the tempting move is to wave the whole thing at the model and call the vagueness "flexibility." A good test: if you can't write down *why* a step needs a model — if the honest answer is "it was easier to prompt than to parse" — that step is probably deterministic work wearing an LLM costume. ## "But my agent's task is whatever the customer asks" Everything so far might read as if this only works when you know the task in advance — my morning routine does the same thing every day, so of course a script can assemble its context. What about an agent facing a customer, where the task is whatever they type? There's no fixed pipeline to make deterministic. The principle survives this; you just have to see where it lives. It was never "the pipeline is fixed." It's the division of labor *at each step*. In a dynamic agent, the model's job is to turn an open-ended ask into a **structured request**: which tool, which parameters, which record, which filter. Interpreting *"can you check why my last order was delayed"* into `get_order(customer_id, latest=true)` is genuinely semantic work — no schema encodes it — so it belongs to the LLM. But once the request is structured, **fulfilling it is deterministic**: the tool runs, the query returns real rows, the file is read. Then the model reasons over context that was *fetched*, not *fabricated*. So the dynamism lives in *deciding what to fetch* — not in the fetching. The customer doesn't dissolve the determinism; they relocate the model's job to the boundary (interpret the ask, route to a source), and everything past that boundary stays deterministic. And this is worth saying plainly: **if your agent already has the LLM choose among deterministic tools, you're applying this principle correctly.** A tool-using agent is the canonical case, not a counterexample. The tools are the deterministic assembly; the model is doing the irreducibly semantic routing. The principle just names *why* that architecture is right — and where it still quietly goes wrong. Because it does still go wrong, even with tools. The line is **generate vs. fetch.** It's fine — necessary — for the model to *decide* what to pull. It's the anti-pattern for the model to *become the source*: recalling the customer's order history from its own weights instead of calling the orders tool, summarizing a policy from memory instead of reading the document, inventing a config value instead of looking it up. The instant the model is the *origin* of a fact rather than the *router* to one, you've traded a deterministic fetch for a hallucination — and in a customer-facing system, that's the fact that lands in an email or a refund. So the discipline for a dynamic agent is narrow and strict: **let the model route freely, but source every fact deterministically.** Route with the LLM; never let it *be* the source of a fact when a source exists. ## Every hallucination is a missing tool All of this only holds if the deterministic source *exists*. Where there's no tool, no query, no schema to fetch from, the model has no choice but to generate — and you're back to inventing facts. Your deterministic layer is only as good as your tool set's coverage, which makes finding the gaps an ongoing job, not a one-time design pass. It reframes a debugging reflex. When an agent hallucinates, the instinct is to tune the prompt. A better first question: *was there a fact here it should have looked up — and does the tool to look it up exist?* A surprising share of hallucinations are **tool gaps in disguise** — the model papering over a hole the architecture left open. So the loop runs continuously: watch where the model is *sourcing* facts instead of *routing* to them; each one is a candidate for a new tool or query; build it; move that fact from generate to fetch. Every tool you add shrinks the surface where fabrication can get in. And the judgment is two-directional — that's the craft. Not every gap deserves a tool. Some residue is genuinely semantic and belongs to the model; over-tooling it is its own failure, brittle and over-fitted. So the standing work is telling apart *"this is a structured fact I just haven't built a source for yet"* (a backlog item) from *"this is irreducibly semantic"* (by design). Your tool set becomes a living map of where the system trusts code versus where it trusts judgment — and keeping that map honest, as the product and its data grow, is a permanent architecture responsibility, not a decision you make once. ## Why determinism at the context layer matters Pushing structure onto code isn't just a cost optimization. It changes the properties of the whole system: - **Reproducibility.** Same input, same context, same setup for the model — every time. When something goes wrong, you can actually reproduce it, because the inputs weren't re-improvised on each run. - **Debuggability.** You can *read* assembled context. It's a file, a query result, a bundle. When the model misbehaves, you inspect exactly what it was handed. Context generated by another model call is a moving target you have to re-run to even see. - **Cost and latency that don't scale with cleverness.** A lookup is a lookup at any volume. You're not paying per-token to answer questions your data already answers. - **A smaller trust surface.** Every LLM call is a place the system can hallucinate. Fewer calls, each with a tighter job, means fewer places for fabrication to enter — and the ones that remain are the ones you *chose*, where semantics genuinely required it. That last point is the crux. Determinism at the context layer is what makes the model layer *trustworthy*. You're not trying to make the LLM perfect; you're shrinking its blast radius to the irreducible semantic core and building everything around it from materials you can verify. ## The architect's move The reframe I'd leave you with: stop thinking of the LLM as the system and the code as glue. Invert it. **The deterministic assembly is the system. The model is one component you call for the part that has no structure.** The prompt is 10%. The 90% is an engineering surface — and most of it, if you look, is already structured. Your job as the architect is to see that structure and refuse to pay a model to reproduce what your code already knows. Generate only what you genuinely can't assemble. Assemble everything else. --- *Built by [Deepak Goyal](https://deepakgoyal.ai). Follow the journey: [GitHub](https://github.com/deepakgoyal-ai) · [LinkedIn](https://linkedin.com/in/deepakgoyal-ai)*