Search for a command to run...
The prompt is only about 10% of what your LLM actually sees. Most of the other 90% is context you assemble — and when your data has structure, you should build it with code, not another LLM call.
The point about hallucinated context being scarier than a hallucinated answer is the one I keep coming back to. When the model fabricates the inputs and downstream steps treat them as ground truth, the failure is silent and much harder to trace. Your BM25-plus-metadata-filter split for exact identifiers matches what I have seen too. How do you decide the cutoff between the structured residue and the genuinely semantic query in practice?
Cadences that surface what's due, habits that flag a slipping streak, and a live HUD that watches the vault — and it ships without a single background cron. SMRITI v2.1.
I'm building an open-source system to interrogate the AI-research firehose — RAG now, a multi-agent researcher by the end, in public over five parts. Part 1 is the foundation: the architecture, the ingestion, and the eval bug that would have shipped meaningless numbers.
A sub-agent that writes to your system will tell you it succeeded. It's reporting what it intended to do, not what actually landed — and the gap is where silent data corruption lives.
You can plan a day around what you can do, or around what you can decide. Until recently those were the same plan, so it never mattered which you meant. Parallel sub-agents pull them apart — and the i
Deepak Goyal
11 posts
Architecture decisions, not opinions. AI systems, system design, and what actually works in production.
The prompt is only about 10% of what your LLM actually sees. Most of the other 90% is context you assemble — and when your data has structure, you should build it with code, not another LLM call.
When people say "prompt engineering," they picture the instruction they typed. But look at what actually arrives in the model's context window on a real request: your system prompt, sure — and then retrieved documents, tool-call results, prior messages, memory, file contents, database rows, schemas. Type it out and the instruction you obsessed over is a thin slice on top of a much larger pile of assembled material.
The prompt is maybe 10% of what the model sees. The other 90% is context you built — and how you built it decides more about the output than any wording tweak.
So the real question isn't "what should I say to the model." It's "how do I assemble the 90%." And that's where I see the same expensive reflex, over and over: teams reach for an LLM to build context that their code already knows.
It's a natural instinct. You have an LLM; it's good with language; so you use it for everything upstream of the "real" call too. A few patterns I keep running into:
Each of these works. That's the trap. It works in the demo, so it ships. Then it runs a thousand times a day and you're paying — in three currencies:
The failure mode that should scare you isn't the model getting an answer wrong. It's the model getting the context wrong, confidently, and the rest of the system treating that fabricated context as ground truth.
Here's the line I've settled on, and it's simple enough to put on a sticky note:
When the data has structure, assemble the context deterministically. Reserve the LLM for the unstructured residue.
Structure is a gift. A schema, a frontmatter block, a database row, a file tree, a tag, a naming convention — every one of those is a place where code can produce the exact right context, the same way, every time, for free. If you can write a query, a template, or a parser that yields the context, you don't need a model to yield it. You need the model only where the input is genuinely unstructured — natural-language judgment, synthesis, semantics that no schema captures.
Most systems have far more structure than they use, and hand the LLM a job that a SELECT or a two-line script would nail deterministically.
I'll make it concrete with systems I actually run.
A pre-assembled context cache. My personal knowledge system (SMRITI — an open-source Obsidian + Claude Code setup) has a lot of standing context an assistant needs on every session: the tag taxonomy, the frontmatter schema, the folder conventions, the area definitions. The naive approach is to let the assistant read five to ten files each time and infer the shape. Instead, a build step assembles all of it into one static file — the "L1 cache" — from the source documents. One deterministic read replaces a scavenger hunt. The content is identical every session because it's assembled by code from structured sources, not re-derived by a model that might read differently on a bad day. The model spends its budget on the actual task, not on rediscovering its own filing system.
Deterministic state bundles. My morning and evening routines used to be "assistant, figure out my day." Now a plain script produces a JSON bundle: today's date, the tasks due, completion counts, which cadences are overdue, project counts by status. All of that is structured — it lives in files and a task API with dates and fields. So a script reads it, and the assistant consumes the bundle instead of inferring it. The judgment-heavy part — "given all this, what should I focus on" — still goes to the model. But the model is reasoning over facts it can trust, because a script produced them, not a summarization pass that might drop a task or invent one.
Retrieval, split by structure. This one is the sharpest. In RAG, there's a reflex to throw an LLM at everything: generate queries, re-rank, summarize. But retrieval quality often hinges on parts that are pure structure. An exact identifier — a function name, an error code, a SKU — is a keyword match; BM25 and a metadata filter find it deterministically and better than a dense model that "sort of" understands the token. Date ranges, categories, source filters — all structured, all deterministic. Reserve the semantic model for the genuinely semantic query ("what do these papers say about reranking"), and let deterministic retrieval own the part that's really a lookup. Stacking an LLM on top of a keyword match doesn't make it smarter; it makes it slower and less certain.
Rule enforcement, tiered. The same split shows up in code-standards enforcement — hundreds of rules to check. The instinct is "it's about code quality, so use the LLM." But a large share of those rules are deterministic: a regex, a config-file assertion, a structural check. Only a minority need real semantic judgment. So the rules are tiered — the mechanical majority run as free, reproducible detectors; only the true-semantic residue is spent on a model call. Same principle: push everything that structure can decide onto deterministic detectors, and let the LLM handle only what's irreducibly semantic.
The pattern is identical across all four: find the structure, let code exploit it, and shrink the model's job to the part that has no structure to exploit.
The useful discipline is a single question you ask at every step of a pipeline:
Does this input have structure that a query, parser, or template can exploit?
Run that question honestly and the residue is almost always smaller than the first design assumed. The architecture work is drawing that boundary well — and it's real work, because the tempting move is to wave the whole thing at the model and call the vagueness "flexibility."
A good test: if you can't write down why a step needs a model — if the honest answer is "it was easier to prompt than to parse" — that step is probably deterministic work wearing an LLM costume.
Everything so far might read as if this only works when you know the task in advance — my morning routine does the same thing every day, so of course a script can assemble its context. What about an agent facing a customer, where the task is whatever they type? There's no fixed pipeline to make deterministic.
The principle survives this; you just have to see where it lives. It was never "the pipeline is fixed." It's the division of labor at each step.
In a dynamic agent, the model's job is to turn an open-ended ask into a structured request: which tool, which parameters, which record, which filter. Interpreting "can you check why my last order was delayed" into get_order(customer_id, latest=true) is genuinely semantic work — no schema encodes it — so it belongs to the LLM. But once the request is structured, fulfilling it is deterministic: the tool runs, the query returns real rows, the file is read. Then the model reasons over context that was fetched, not fabricated.
So the dynamism lives in deciding what to fetch — not in the fetching. The customer doesn't dissolve the determinism; they relocate the model's job to the boundary (interpret the ask, route to a source), and everything past that boundary stays deterministic.
And this is worth saying plainly: if your agent already has the LLM choose among deterministic tools, you're applying this principle correctly. A tool-using agent is the canonical case, not a counterexample. The tools are the deterministic assembly; the model is doing the irreducibly semantic routing. The principle just names why that architecture is right — and where it still quietly goes wrong.
Because it does still go wrong, even with tools. The line is generate vs. fetch. It's fine — necessary — for the model to decide what to pull. It's the anti-pattern for the model to become the source: recalling the customer's order history from its own weights instead of calling the orders tool, summarizing a policy from memory instead of reading the document, inventing a config value instead of looking it up. The instant the model is the origin of a fact rather than the router to one, you've traded a deterministic fetch for a hallucination — and in a customer-facing system, that's the fact that lands in an email or a refund.
So the discipline for a dynamic agent is narrow and strict: let the model route freely, but source every fact deterministically. Route with the LLM; never let it be the source of a fact when a source exists.
All of this only holds if the deterministic source exists. Where there's no tool, no query, no schema to fetch from, the model has no choice but to generate — and you're back to inventing facts. Your deterministic layer is only as good as your tool set's coverage, which makes finding the gaps an ongoing job, not a one-time design pass.
It reframes a debugging reflex. When an agent hallucinates, the instinct is to tune the prompt. A better first question: was there a fact here it should have looked up — and does the tool to look it up exist? A surprising share of hallucinations are tool gaps in disguise — the model papering over a hole the architecture left open.
So the loop runs continuously: watch where the model is sourcing facts instead of routing to them; each one is a candidate for a new tool or query; build it; move that fact from generate to fetch. Every tool you add shrinks the surface where fabrication can get in.
And the judgment is two-directional — that's the craft. Not every gap deserves a tool. Some residue is genuinely semantic and belongs to the model; over-tooling it is its own failure, brittle and over-fitted. So the standing work is telling apart "this is a structured fact I just haven't built a source for yet" (a backlog item) from "this is irreducibly semantic" (by design). Your tool set becomes a living map of where the system trusts code versus where it trusts judgment — and keeping that map honest, as the product and its data grow, is a permanent architecture responsibility, not a decision you make once.
Pushing structure onto code isn't just a cost optimization. It changes the properties of the whole system:
That last point is the crux. Determinism at the context layer is what makes the model layer trustworthy. You're not trying to make the LLM perfect; you're shrinking its blast radius to the irreducible semantic core and building everything around it from materials you can verify.
The reframe I'd leave you with: stop thinking of the LLM as the system and the code as glue. Invert it. The deterministic assembly is the system. The model is one component you call for the part that has no structure.
The prompt is 10%. The 90% is an engineering surface — and most of it, if you look, is already structured. Your job as the architect is to see that structure and refuse to pay a model to reproduce what your code already knows.
Generate only what you genuinely can't assemble. Assemble everything else.
Built by Deepak Goyal. Follow the journey: GitHub · LinkedIn
