The glamorous layer is the agent. The load-bearing one is the boring shell script underneath — and that's where most of your reliability actually lives.

The highest-leverage code I shipped this month wasn't a prompt. It wasn't an agent. It was a handful of boring shell scripts.

They sit under my second brain — the system I wrote about recently, the one that stopped needing me to schedule it. That post was about what the automation does. This one is about the part that actually carries it, which I glossed over at the time: not the AI. The scripts.

The two kinds of work in every AI system

Every AI automation is really doing two different jobs, and we tend to blur them.

One is judgment: decide which thing matters, draft the reply, synthesize the week into a paragraph. That needs a model — it's taste, not logic.

The other is deterministic: fetch the data, parse it, dedupe it, mutate a file, check a condition. That needs no brain at all. And yet the reflex, once you have a capable model in the loop, is to hand it that work too — because it can do it. It can. It'll also cost you tokens and hand back a slightly different answer every run.

The move that keeps earning its keep is dull to state and hard to remember: push everything deterministic down into a script, and spend the model only on the judgment.

What the split actually looks like

I built a small system that surfaces conversations worth joining — a morning shortlist of threads where I genuinely have something to add. It has exactly two layers.

A shell script does the deterministic half: hit the API, pull candidate threads, keyword-match them against my areas, drop anything I've seen before. Some JSON parsing, a grep, a dedupe set. Nothing in there wants a brain.

An agent does the judgment half: read those candidates, pick the few I can actually help with, and draft a reply in my voice.

If I'd let the model do the fetching and filtering too, I'd be paying tokens to do a job grep does perfectly — and getting a slightly different shortlist each morning. Instead the mechanical part is fixed, fast, and free, and the model spends its whole budget on the part that's genuinely a decision.

A script that breaks tells you it broke

Here's the part that settled it for me.

The morning after I wired it up, the shortlist came back empty. My instinct was to distrust the whole thing. But because the fetch layer is a script, I could ask it exactly what happened in one line:

curl -s -o /dev/null -w "%{http_code}\n" https://…
→ 403

The free API had started refusing unauthenticated requests. Thirty seconds to diagnose, because the failure was a number, not a mood.

Now put that same fetch inside a model loop. It hits the same wall — and then does what models do: smooths over the gap, returns a plausible empty result, and I trust it. Deterministic systems fail loudly and specifically. Probabilistic ones fail quietly and expensively. A script that breaks tells you it broke; that is worth far more than it sounds.

The cheapest guardrail I own is a grep

One more, because it's my favourite.

I keep a public, open-source version of my system, built by copying pieces out of my private one and stripping the private bits. A single leaked name or path in a public repo is the sort of mistake you can't quietly undo. So before I tag a release, a script greps the staged files for anything private — names, absolute paths, tokens — and exits non-zero if it finds one. A spellcheck for secrets. Zero tokens, runs in a second, and it cannot have an off day.

I could ask a model, "is anything private in here?" It would be slower, cost tokens, and — the part that matters — it could miss differently each time. The grep can only catch what's on its list, too; it's not magic. But it catches those every single time, and the day something slips through a new pattern, I add one line and it's caught on every release after. That's the trade I want in a safety gate: it fails the same way twice, so I only have to fix it once.

Why it's the highest ROI, not just the cheapest

It's tempting to say the win is cost, and it is. But cost is the bonus. The reason is three properties you get from code and cannot get from a model: it's reliable (same input, same output, forever — no drift, no temperature), it's testable (you can unit-test a script; you cannot unit-test a vibe), and it fails visibly (an exit code, not a confident wrong answer). Write it once, run it every morning, at zero marginal cost. That's the curve you want underneath the expensive, probabilistic layer — not on top of it.

Where the line sits — and it cuts both ways

The skill isn't "script everything." It's knowing where the boundary is.

Fetching, parsing, deduping, mutating files, checking a condition — that's below the line, and it belongs in a script. Deciding which thread deserves your voice, drafting the reply, reading a messy week and calling the theme — that's above it, and it belongs to the model.

Push the line as far toward script as the work honestly allows, and not one step past. Script the judgment and you get brittle rules that shatter on the first case you didn't foresee. Hand determinism to the model and you get expensive nondeterminism. They're the same mistake, pointed in opposite directions.

The load-bearing layer

The layer that ends up in the demo is the agent. The layer that keeps the thing standing is the script underneath it — and a surprising amount of your reliability lives down there, unglamorous and unmentioned.

If you're building anything with AI this week, the highest-leverage hour probably isn't a sharper prompt. It's taking one deterministic thing your model is quietly doing, and moving it into a small, dull script that will never surprise you.

Let the model do the thinking. Everything else is a job for something that shows up, does the boring thing the same way every time, and tells you plainly when it can't.

The prompt is only about 10% of what your LLM actually sees. Most of the other 90% is context you assemble — and when your data has structure, you should build it with code, not another LLM call.

The 90% nobody edits

When people say "prompt engineering," they picture the instruction they typed. But look at what actually arrives in the model's context window on a real request: your system prompt, sure — and then retrieved documents, tool-call results, prior messages, memory, file contents, database rows, schemas. Type it out and the instruction you obsessed over is a thin slice on top of a much larger pile of assembled material.

The prompt is maybe 10% of what the model sees. The other 90% is context you built — and how you built it decides more about the output than any wording tweak.

So the real question isn't "what should I say to the model." It's "how do I assemble the 90%." And that's where I see the same expensive reflex, over and over: teams reach for an LLM to build context that their code already knows.

The reflex: generate what you could assemble

It's a natural instinct. You have an LLM; it's good with language; so you use it for everything upstream of the "real" call too. A few patterns I keep running into:

• LLM-as-summarizer for content that's already structured — asking a model to "summarize the project status" from files whose frontmatter already has the status, the owner, the due date.
• LLM-as-router — a call to decide "which of these five categories does this belong to" when a lookup table or a label already answers it.
• LLM-as-extractor — parsing a field out of a document with a prompt when the document is JSON, or the field is a column.
• LLM-as-selector — "figure out which of my 200 notes are relevant" when the notes carry tags, links, and dates you could filter on directly.

Each of these works. That's the trap. It works in the demo, so it ships. Then it runs a thousand times a day and you're paying — in three currencies:

1. Money and latency. Every one of those is a network round-trip and a token bill for something a query would return in a millisecond for free.
2. Non-determinism. Same input, slightly different context tomorrow. Your pipeline develops a shimmer. Reproducing a bad output becomes archaeology.
3. Hallucinated context. This is the quiet one. When you ask a model to generate the context for the next step, it can invent. Now you're not just risking a hallucinated answer — you've hallucinated the inputs, and everything downstream trusts them.

The failure mode that should scare you isn't the model getting an answer wrong. It's the model getting the context wrong, confidently, and the rest of the system treating that fabricated context as ground truth.

The principle

Here's the line I've settled on, and it's simple enough to put on a sticky note:

When the data has structure, assemble the context deterministically. Reserve the LLM for the unstructured residue.

Structure is a gift. A schema, a frontmatter block, a database row, a file tree, a tag, a naming convention — every one of those is a place where code can produce the exact right context, the same way, every time, for free. If you can write a query, a template, or a parser that yields the context, you don't need a model to yield it. You need the model only where the input is genuinely unstructured — natural-language judgment, synthesis, semantics that no schema captures.

Most systems have far more structure than they use, and hand the LLM a job that a SELECT or a two-line script would nail deterministically.

What this looks like in practice

I'll make it concrete with systems I actually run.

A pre-assembled context cache. My personal knowledge system (SMRITI — an open-source Obsidian + Claude Code setup) has a lot of standing context an assistant needs on every session: the tag taxonomy, the frontmatter schema, the folder conventions, the area definitions. The naive approach is to let the assistant read five to ten files each time and infer the shape. Instead, a build step assembles all of it into one static file — the "L1 cache" — from the source documents. One deterministic read replaces a scavenger hunt. The content is identical every session because it's assembled by code from structured sources, not re-derived by a model that might read differently on a bad day. The model spends its budget on the actual task, not on rediscovering its own filing system.

Deterministic state bundles. My morning and evening routines used to be "assistant, figure out my day." Now a plain script produces a JSON bundle: today's date, the tasks due, completion counts, which cadences are overdue, project counts by status. All of that is structured — it lives in files and a task API with dates and fields. So a script reads it, and the assistant consumes the bundle instead of inferring it. The judgment-heavy part — "given all this, what should I focus on" — still goes to the model. But the model is reasoning over facts it can trust, because a script produced them, not a summarization pass that might drop a task or invent one.

Retrieval, split by structure. This one is the sharpest. In RAG, there's a reflex to throw an LLM at everything: generate queries, re-rank, summarize. But retrieval quality often hinges on parts that are pure structure. An exact identifier — a function name, an error code, a SKU — is a keyword match; BM25 and a metadata filter find it deterministically and better than a dense model that "sort of" understands the token. Date ranges, categories, source filters — all structured, all deterministic. Reserve the semantic model for the genuinely semantic query ("what do these papers say about reranking"), and let deterministic retrieval own the part that's really a lookup. Stacking an LLM on top of a keyword match doesn't make it smarter; it makes it slower and less certain.

Rule enforcement, tiered. The same split shows up in code-standards enforcement — hundreds of rules to check. The instinct is "it's about code quality, so use the LLM." But a large share of those rules are deterministic: a regex, a config-file assertion, a structural check. Only a minority need real semantic judgment. So the rules are tiered — the mechanical majority run as free, reproducible detectors; only the true-semantic residue is spent on a model call. Same principle: push everything that structure can decide onto deterministic detectors, and let the LLM handle only what's irreducibly semantic.

The pattern is identical across all four: find the structure, let code exploit it, and shrink the model's job to the part that has no structure to exploit.

Drawing the line

The useful discipline is a single question you ask at every step of a pipeline:

Does this input have structure that a query, parser, or template can exploit?

• Yes → assemble it with code. Schema fields, tags, dates, IDs, file paths, config, table rows, anything with a grammar. Deterministic, free, reproducible.
• No → this is the residue. Natural-language meaning, open-ended synthesis, judgment that no field encodes. This is what the LLM is for.

Run that question honestly and the residue is almost always smaller than the first design assumed. The architecture work is drawing that boundary well — and it's real work, because the tempting move is to wave the whole thing at the model and call the vagueness "flexibility."

A good test: if you can't write down why a step needs a model — if the honest answer is "it was easier to prompt than to parse" — that step is probably deterministic work wearing an LLM costume.

"But my agent's task is whatever the customer asks"

Everything so far might read as if this only works when you know the task in advance — my morning routine does the same thing every day, so of course a script can assemble its context. What about an agent facing a customer, where the task is whatever they type? There's no fixed pipeline to make deterministic.

The principle survives this; you just have to see where it lives. It was never "the pipeline is fixed." It's the division of labor at each step.

In a dynamic agent, the model's job is to turn an open-ended ask into a structured request: which tool, which parameters, which record, which filter. Interpreting "can you check why my last order was delayed" into get_order(customer_id, latest=true) is genuinely semantic work — no schema encodes it — so it belongs to the LLM. But once the request is structured, fulfilling it is deterministic: the tool runs, the query returns real rows, the file is read. Then the model reasons over context that was fetched, not fabricated.

So the dynamism lives in deciding what to fetch — not in the fetching. The customer doesn't dissolve the determinism; they relocate the model's job to the boundary (interpret the ask, route to a source), and everything past that boundary stays deterministic.

And this is worth saying plainly: if your agent already has the LLM choose among deterministic tools, you're applying this principle correctly. A tool-using agent is the canonical case, not a counterexample. The tools are the deterministic assembly; the model is doing the irreducibly semantic routing. The principle just names why that architecture is right — and where it still quietly goes wrong.

Because it does still go wrong, even with tools. The line is generate vs. fetch. It's fine — necessary — for the model to decide what to pull. It's the anti-pattern for the model to become the source: recalling the customer's order history from its own weights instead of calling the orders tool, summarizing a policy from memory instead of reading the document, inventing a config value instead of looking it up. The instant the model is the origin of a fact rather than the router to one, you've traded a deterministic fetch for a hallucination — and in a customer-facing system, that's the fact that lands in an email or a refund.

So the discipline for a dynamic agent is narrow and strict: let the model route freely, but source every fact deterministically. Route with the LLM; never let it be the source of a fact when a source exists.

Every hallucination is a missing tool

All of this only holds if the deterministic source exists. Where there's no tool, no query, no schema to fetch from, the model has no choice but to generate — and you're back to inventing facts. Your deterministic layer is only as good as your tool set's coverage, which makes finding the gaps an ongoing job, not a one-time design pass.

It reframes a debugging reflex. When an agent hallucinates, the instinct is to tune the prompt. A better first question: was there a fact here it should have looked up — and does the tool to look it up exist? A surprising share of hallucinations are tool gaps in disguise — the model papering over a hole the architecture left open.

So the loop runs continuously: watch where the model is sourcing facts instead of routing to them; each one is a candidate for a new tool or query; build it; move that fact from generate to fetch. Every tool you add shrinks the surface where fabrication can get in.

And the judgment is two-directional — that's the craft. Not every gap deserves a tool. Some residue is genuinely semantic and belongs to the model; over-tooling it is its own failure, brittle and over-fitted. So the standing work is telling apart "this is a structured fact I just haven't built a source for yet" (a backlog item) from "this is irreducibly semantic" (by design). Your tool set becomes a living map of where the system trusts code versus where it trusts judgment — and keeping that map honest, as the product and its data grow, is a permanent architecture responsibility, not a decision you make once.

Why determinism at the context layer matters

Pushing structure onto code isn't just a cost optimization. It changes the properties of the whole system:

• Reproducibility. Same input, same context, same setup for the model — every time. When something goes wrong, you can actually reproduce it, because the inputs weren't re-improvised on each run.
• Debuggability. You can read assembled context. It's a file, a query result, a bundle. When the model misbehaves, you inspect exactly what it was handed. Context generated by another model call is a moving target you have to re-run to even see.
• Cost and latency that don't scale with cleverness. A lookup is a lookup at any volume. You're not paying per-token to answer questions your data already answers.
• A smaller trust surface. Every LLM call is a place the system can hallucinate. Fewer calls, each with a tighter job, means fewer places for fabrication to enter — and the ones that remain are the ones you chose, where semantics genuinely required it.

That last point is the crux. Determinism at the context layer is what makes the model layer trustworthy. You're not trying to make the LLM perfect; you're shrinking its blast radius to the irreducible semantic core and building everything around it from materials you can verify.

The architect's move

The reframe I'd leave you with: stop thinking of the LLM as the system and the code as glue. Invert it. The deterministic assembly is the system. The model is one component you call for the part that has no structure.

The prompt is 10%. The 90% is an engineering surface — and most of it, if you look, is already structured. Your job as the architect is to see that structure and refuse to pay a model to reproduce what your code already knows.

Generate only what you genuinely can't assemble. Assemble everything else.

Built by Deepak Goyal. Follow the journey: GitHub · LinkedIn

What I'm building, and why

Nobody can keep up with AI research anymore. On ArXiv — the open-access preprint server where nearly all AI/ML work lands first — the three machine-learning categories I care about (cs.AI, cs.CL, and cs.IR) publish more papers in a week than anyone can read in a month. The tooling for interrogating that firehose — "what do the last hundred papers say about reranking, and can I trust the answer?" — barely exists outside closed products.

So I'm building one, in the open. A system that ingests a pinned corpus of ArXiv papers and answers real questions against it: retrieval-augmented generation (RAG) today, a multi-agent researcher-critic-synthesizer by the end. The code is MIT-licensed on GitHub, the write-ups live at deepakgoyal.ai, and it's reproducible down to the exact paper. It ships over five parts — each a working release and a post like this one. Not a tutorial, and not a learning diary: an architect building a real system and showing the decisions.

This post is Part 1: the foundation — the corpus, the ingestion pipeline, the retrieval architecture, and the first evaluation framework. It's the least glamorous slice and the one everything else stands on.

Here's the arc, so you know what you're reading toward:

1. Part 1 (this post) — Foundation. Pinned corpus, retrieval pipeline, honest eval, FastAPI service.
2. Part 2 — RAG to agent. Tool use: summarize a paper, discover citations, find related work.
3. Part 3 — MCP server. Expose the agent's tools so Claude Desktop/Code can call them directly.
4. Part 4 — ReAct workflows. Multi-step, plan-and-execute research tasks.
5. Part 5 — Multi-agent. A researcher, a synthesizer, and a critic collaborating.

And one rule runs through all five, the whole reason I write these the way I do: measurement before claim. Every comparison I publish shows its methodology, its sample size, its cost, and a command you can run to reproduce it. No "it's better" without the eval that says so.

Which is exactly why Part 1 has to open with a confession: my first eval was measuring nothing.

The eval that was lying to me

I built a retrieval evaluation, ran it, and got a clean table of numbers. Then I looked closer and realized the metric was scoring itself — the way I'd defined "relevant" was rigged, by construction, to make retrieval look good. I was one commit from publishing a table that looked rigorous and meant nothing. (The exact mechanism is in the retrieval section below, where the numbers live.)

I cut the broken metrics and kept only what survives scrutiny. That's what this post is really about: the interesting part of building RAG isn't the pipeline, which is largely solved — it's knowing whether the thing works, and being honest when your own measurement can't tell you.

The rest is the foundation, in the order I'd build it again: the corpus, the pipeline, and the two evals — one I threw out, one I'll stand behind.

Why ArXiv, why 100 papers, why free-first

ArXiv because the corpus is stable (papers don't change under you), acquisition is trivial, the license is friendly, and the content matches the audience — AI/ML research in cs.AI, cs.CL, cs.IR.

100 papers because it's large enough to escape toy-corpus saturation — where every query trivially retrieves the same handful of chunks — without burning a rate-limit budget on volume I couldn't yet justify. The corpus is pinned: 100 papers, 11,187 chunks. I'll scale to a few hundred only when a gold-labelled eval shows a recall ceiling worth chasing, not before.

Free-first because most of this workload doesn't need a paid model. BGE-small embeddings run locally, the ChromaDB vector store is local, the cross-encoder reranker is local, and the default answer model is a free Groq-hosted Llama-3.3-70B. Paid models come out only for documented A/B comparisons on small samples. Total project budget: ~$15.

That's a decision, not a limitation. A RAG pipeline that only works when every call hits a frontier API isn't a pipeline — it's a bill.

The pipeline — ingestion decides whether anything downstream is trustworthy

The whole thing is one line of dataflow:

ingest → chunk + embed → ChromaDB → retrieve → (rerank / fuse) → generate → cite

exposed over FastAPI (/search, /ask, /chat, /ingest, /health).

Ingestion is the unglamorous part, and it's where trust is won or lost. It hits the ArXiv API with a 3-second politeness delay and automatic retries, and it's idempotent per paper — an interrupted run resumes instead of restarting. Any PDF that won't parse is logged in the run summary and skipped, never silently dropped or left to crash the batch. The output is data/manifest.jsonl — a committed record that pins every paper by its ArXiv ID. That file is the corpus. Anyone can rebuild the identical 100 papers from it.

Those first two items aren't decoration: ArXiv rate-limits aggressive clients, and an early impatient run earned a batch of 429s and 503s and downloaded nothing. Slowing down and checkpointing fixed it — the pinned rebuild then pulled all 100 papers with zero parse failures.

Retrieval is one idiom, repeated. Every retriever is the same shape — a retrieve(query, top_k) -> list[dict] closure over one consistent chunk dictionary — so a new strategy slots in without touching the callers. Six ship in Part 1, and since these names recur in the results below, here's each in a line:

• dense — embed the query and return the nearest chunks by vector similarity (semantic match);
• BM25 — classic keyword scoring; strong exactly where dense is weak, on rare terms and proper nouns;
• hybrid — run dense and BM25, then fuse the two rankings (see RRF, next);
• cross-encoder rerank — take a candidate pool and re-score it with a model that reads the query and chunk together, rather than comparing pre-computed vectors;
• multi-query — have an LLM rephrase the question into several variants, retrieve for each, and fuse;
• HyDE — have an LLM draft a hypothetical answer first, then retrieve against that answer instead of the bare question.

One design note I'd repeat on any RAG build: implement fusion once. A single Reciprocal Rank Fusion (RRF) routine — rank each list, score every item by 1 / (60 + rank) (60 is RRF's smoothing constant, unrelated to the top-k cutoff), sum across lists — serves both hybrid retrieval (fuse 2 lists: dense + BM25) and multi-query (fuse N lists: one per variant). They look like different features; they're the same operation. Writing it twice is how the two copies drift.

The conversational path is history → LLM rewrite → retrieve → answer, with a bypass_rewrite escape hatch for the failure mode where the rewrite over-constrains a follow-up and retrieval returns nothing.

Measurement — what's honest and what isn't

Here's where most RAG write-ups quietly cheat. Two evals ran on this corpus. Only one produces a number I'll stand behind — and before either table, here's exactly how each number is computed, because a metric you can't reproduce is just a vibe with a decimal point.

How the numbers are computed

Retrieval — category-precision@k. Per question: take the top k retrieved chunks (k=5), look up the source paper of each, and check whether that paper's primary ArXiv category is one the question expects (its source_hint, e.g. cs.IR cs.CL). The score is the fraction of the five that match; the reported figure averages that over every question with a usable hint (27 of the 33). It answers one narrow question — "did retrieval stay in the right topic?" — and nothing more.

Generation — claim-level faithfulness. Faithfulness measures how much of an answer is grounded in the retrieved context. A fixed judge model scores it in two steps, at temperature 0:

1. Extract — the judge reads the answer and lists every atomic factual claim it makes.
2. Verify — each claim is checked independently against the retrieved chunks and marked supported, unsupported, or not_found.

The question's score is supported ÷ total claims. So if an answer makes four claims and three trace to a sentence in the retrieved context, it scores 0.75 — whether or not the fourth claim happens to be true in the world. That distinction is the whole point: faithfulness asks "did the model stick to its sources," not "is the model right." A confident, plausible sentence with no support in the context is exactly what it's built to catch — because in RAG, that's the shape a hallucination takes.

Each model's faithfulness is averaged across the questions, with a 95% bootstrap confidence interval (2,000 resamples, fixed seed) so the small sample size is visible in the number rather than hidden behind it. When two models' intervals don't overlap, the difference is unlikely to be sampling noise; when they do, treat the gap as unproven.

One query, end to end

Abstract metrics are easy to wave at, so here are two real questions from the eval set, run through the real pipeline.

A retrieval miss. Ask "What is HyDE, and when does it fail?" and the hybrid retriever's top 5 chunks come back as:

Not one is about HyDE. The corpus is 100 recent papers and none of them defines it, so retrieval stays vaguely in the machine-learning neighborhood and misses. Now watch what the proxy does with that miss: the question's hint is cs.IR cs.CL, so category-precision counts the single cs.CL paper (#5) as a hit and the rest as misses — a score of 0.2. But #5 isn't about HyDE either, and the cs.AI papers it scored as misses might be more relevant or less — the metric can't tell. Whether it lands at 0.2 or 0.8, the number is blind to the one thing that matters: did retrieval surface a chunk that actually answers the question. "In the right topic" is all it can ever see.

A perfect faithfulness score for a non-answer. Now a question the corpus only mentions in passing — "How does Reciprocal Rank Fusion combine rankings?" Given the retrieved context, Claude Haiku answers:

"…the documents do not provide a detailed explanation of how RRF actually works mathematically or algorithmically… I don't know the detailed methodology based on the provided context."

The judge extracts five atomic claims from that reply (e.g., "RRF merges ranked lists from multiple retrieval channels," "the specific formula is not explained in the provided context") and finds all five supported by the retrieved chunks:

faithfulness = 5 / 5 = 1.00.

A perfect score — for an answer that never answered the question. And that's the metric working correctly: the retrieved chunks mentioned RRF but never gave its formula, so the model declined instead of inventing one, and every claim it did make was grounded. Faithfulness rewards not hallucinating; it says nothing about whether the question got a useful answer. Which is exactly why it's one axis of quality — the one that catches confident fabrication — and not the whole of it. (Getting the right chunk in front of the model in the first place is the retrieval problem above, and the reason Part 2 starts with gold labels.)

The retrieval eval (the honesty lesson)

Here's the full mechanism behind the bug at the top of this post. My 33-question eval set carries topical source_hint categories but no gold chunk or document labels — nobody has marked "for this question, chunk X is the right answer." My first pass papered over that by deriving each question's "relevant" set from the retrieved results themselves — relevant ⊆ retrieved by construction — so recall was pinned at 1.0-or-0.0 and "precision" only measured category purity. Any recall or MRR computed that way grades the retriever against its own output. That leaves category-precision as the one signal I can defend, and even it only measures topical purity — not whether the right chunk was found.

n = 27 scored of 33 questions · coarse topical-purity proxy — NOT relevance recall.

Read that as a diagnostic, not a leaderboard. Reranking (config B) scores lowest — but read that carefully rather than as "reranking is bad." A cross-encoder pulls the single most-relevant chunk to the top; if that chunk happens to sit in a neighbouring category, category-precision counts it as a miss. So a low score here can mean the reranker did its job, not that it failed — which is the whole problem: none of these numbers measure whether the right chunk was found, so none is a verdict on retrieval quality. Publishing 0.27 as a retrieval-quality score would be exactly the lie I caught myself in. Gold doc-level labels are the next eval task (Part 2); publishable retrieval claims wait for them.

The generation A/B (the number I'll stand behind)

This one is honest because the metric doesn't score itself. Hold retrieval constant — the same retrieved context per question — vary only the answer model, and score claim-level faithfulness with one fixed judge (GPT-4o-mini). Five models, three provider families, across two size tiers, on a 20-question sample — the first 20 of the 33, a fixed deterministic slice, chosen only to cap paid-model spend:

n = 20 · judge: GPT-4o-mini (fixed) · faithfulness = supported claims ÷ total claims · 95% bootstrap CI. Latency is deliberately omitted — cached answers contaminate the timings, so a clean latency pass is a separate cold-cache run.

Three things this table says, and one it doesn't.

Claude is in a different grounding regime. Both Anthropic models cluster at 0.83, and their confidence intervals sit clear of every other model's — this is the one gap in the table that isn't sampling noise. Given identical retrieved context, Claude confines its answer to what that context supports far more consistently than the GPT or Gemini models do.

Bigger did not mean more faithful. Sonnet (0.83) is a dead heat with Haiku (0.83), and GPT-4o's point estimate (0.20) sits below GPT-4o-mini's (0.28) — though those two overlap within the sample, so read that as "no gain," not "GPT-4o is worse." Either way the direction is clear: whatever the flagship tiers buy — reasoning depth, breadth, polish — it isn't grounding on this task. If faithfulness is the axis you care about, the cheap tier is the buy; paying up bought no measurable grounding here.

The likely mechanism is the interesting part. Faithfulness measures grounding, not correctness. The GPT models tend to answer from what they already "know" about retrieval and RAG — often perfectly reasonable answers — but any claim the model supplies from its own training that the retrieved chunks don't contain scores as unsupported. Claude more often stays inside the provided context — the RRF example, where it flagged what the context didn't cover instead of filling the gap from memory, is that discipline in miniature. So read the table precisely: it does not say GPT-4o is a worse model. It says GPT-4o is a less grounded one — and for a RAG answerer meant to cite its sources, grounding is exactly the property you're paying for.

And it isn't Claude just hedging. The sharp objection to this whole table: a model that declines to answer trivially scores 100% faithful — the RRF non-answer above is exactly that — so maybe Claude wins by saying less. It doesn't. Counting the claims the judge extracts per answer, the Claude models make more of them, not fewer: about 6 (Haiku) and 8 (Sonnet) per answer, versus roughly 3 for GPT-4o-mini, GPT-4o, and Gemini. Claude is asserting twice as much and still grounding a higher share of it — saying more and staying honest, not saying less to protect the ratio.

And the one thing it doesn't say — the judge isn't playing favorites. GPT-4o-mini grades every answer, including its own family's, and it ranks OpenAI last. Whatever bias a single LLM judge carries, self-flattery isn't it here.

The caveats, stated plainly, because they're the point:

• The judge is an LLM proxy, not ground truth. A single fixed judge keeps the scores comparable, but it's still a model grading models — it can misjudge a claim's support, and it inherits GPT-4o-mini's own blind spots. A second independent judge and some human spot-checks are on the list before I'd treat any single score as settled.
• The free production default isn't in this table. The pipeline's real answerer is the free Groq Llama-3.3-70B; this A/B benchmarks paid alternatives to decide whether paying buys enough grounding to matter. Measuring the free default head-to-head needs a configured Groq key in the eval environment — it's the obvious next baseline, not a result I'm hiding.
• Sample size is 20, not 3,300. The bootstrap intervals are wide on purpose. Non-overlapping intervals (Claude vs the rest) are real signal; overlapping ones (Haiku vs Sonnet, or the three trailing models against each other) are "not yet distinguished," not "the same."

Reproducibility and cost

The corpus rebuilds from the committed manifest:

uv sync
uv run python scripts/run_ingest.py --max 100   # or rebuild the pinned IDs from data/manifest.jsonl
uv run python scripts/build_corpus.py            # 100 papers → 11,187 chunks → ChromaDB (~115 MB)
uv run python tests/eval/run_ab.py               # retrieval A/B (configs A–E)
uv run python tests/eval/run_llm_ab.py --n 20    # generation A/B (5 models)

One honesty note on the ingest line: ArXiv's "latest N" query drifts by the day, so --max 100 today fetches today's newest 100, not my pinned set. The reproducible artifact is data/manifest.jsonl — it pins all 100 ArXiv IDs, and the build downloads exactly those by ID. The query bootstraps the corpus; the manifest is the corpus.

Cost. The corpus build and the retrieval A/B run on free and local models — local BGE-small embeddings, local cross-encoder, ChromaDB, and the free Groq Llama default for the query-rewrite and HyDE steps → \(0. The generation A/B is the only paid run: five hosted models answering 20 questions each, plus a GPT-4o-mini judge extracting and verifying claims — an estimated ~\)0.30 (the repo doesn't meter tokens, so treat that as order-of-magnitude, not metered; GPT-4o and Claude Sonnet are the only non-trivial line items). Every LLM call is disk-cached, so re-running the identical eval is free. Running total stays inside the ~$15 project budget.

What's next

Part 2 turns this from a pipeline into an agent — tool-using, able to summarize a paper, discover citations, and find related work on its own. And before any retrieval-quality claim gets published, the eval gets gold doc-level labels, so the honest-but-limited category proxy above can finally become a real precision number.

The pattern I'm trying to hold to across all five parts: show the architecture, show the eval, show the cost — and when the eval can't back the claim, say so out loud instead of shipping the pretty table.

I'm a Technical Architect building AI-enabled systems. I write about what I build — the architecture and measurement decisions under the hood, not the model hype. The full project is open-source.

If you've caught your own eval measuring the wrong thing, I'd like to hear how. Find me on X @deepakgoyal_ai or LinkedIn.

Nothing in the agent looked broken from the inside. It had decided to close them, issued what it believed were the right calls, and then did what language models are trained to do: wrote a confident, helpful account of the work. The account described its intention. Nothing in its loop ever went back to check whether reality agreed.

That gap — between what a sub-agent intends and what actually lands in your system — is where silent failures breed. And you can't prompt your way out of it.

Why the narrative and the outcome drift apart

A language model generates a plausible continuation of its context. When you hand a sub-agent a job and it runs a sequence of tool calls, its final "here's what I did" message is another plausible continuation — a summary of the trajectory it took, weighted toward the outcome it was aiming for.

That's fine when the worst case is a slightly-off answer. It is not fine when the agent is mutating state — writing to a database, closing tickets, moving files, calling a third-party API. There, the summary is a claim about the world, and the model has no privileged access to the world. It has access to its own transcript.

A loud error shows up in that transcript. But the dangerous failures don't error. A call that silently no-ops, drops a field, or half-applies still returns a success-shaped response — and from inside the transcript that's indistinguishable from a call that did exactly what was asked. So the agent reports success, because from where it's standing, success is the most plausible thing that happened.

The uncomfortable part: the better your model is at writing helpful narratives, the more convincing these false reports get.

Four silent failures from one real agent

This isn't hypothetical. I run a personal "second brain" system where a sync agent reconciles my task manager with my notes — creating, updating, and closing tasks across a boundary. Hardening that one agent surfaced four distinct failure classes:

• The dropped prefix. Tasks were supposed to be written with a structured [<project> <id>] title prefix so they could be traced back. The agent "created" them and reported success — but the prefix was silently missing on a batch, breaking the link with no error anywhere.
• The default lane. A required field that assigns each task to its correct section was omitted on some writes. The tasks landed — just dumped into the default bucket instead of where they belonged. Fully "successful," entirely misfiled.
• The phantom close. The six-tasks-closed story above. The agent asserted the close happened; the source of truth said all six were still active.
• The mislabeled multitude. A labeling step inferred each task's category from its folder location. One bug in that inference was a single run away from stamping the wrong label onto 47 tasks — a confident, bulk, wrong mutation, caught only because I read the plan back before it fired.

Every one of these produced a clean success report. Not one threw an error. If I'd trusted the narrative, I'd have accumulated silent drift for weeks before an incident forced me to notice.

The fix has one shape: read your own write back

All four fixes converged on the same structural move — and none of them was "write a better prompt."

Close the loop against the source of truth. After a mutation, the agent (or the orchestrator above it) re-reads the thing it just wrote — through the actual system API, not its own memory — and asserts the post-state matches what it expected. Closed a task? Fetch it back and confirm it reads as closed. Set a field? Read the record and confirm the field is there. The narrative is no longer evidence; the read-back is.

Three concrete disciplines fall out of that:

1. Pre-flight refusal. If a required field is absent before the write, the agent refuses and surfaces it — rather than writing a malformed record and calling it done. Fail loudly at the boundary, not silently after it.
2. Post-write validation. Every state-changing call is followed by a read-back assertion against the source of truth. No assertion, no "success."
3. Explicit contracts. Schema maps and label namespaces are written down and matched against before the write — so "infer it from context" never silently produces the wrong structure.

Notice these are all structural checks that live outside the model's judgment. That's the point. You are not asking the agent to be more careful. You are removing its self-report from the trust path entirely.

Where this applies — and where it's overkill

This discipline earns its cost precisely when a wrong outcome is silent and durable:

• Any sub-agent that writes to an external system — a database, a task manager, a repo, a third-party API.
• Bulk operations where the failure mode is quiet drift rather than a loud crash — sync agents, migration agents, cleanup passes.
• Designs where the orchestrator can't cheaply diff before-and-after on its own.

It's overkill in the opposite cases:

• Read-only agents — search, summarize, scan — where the worst case is a bad answer the next step will catch, not corrupted state.
• Single-call write paths where a human sees the result immediately in the next turn and would spot a failure anyway.

The question that replaces "is the model smart enough?"

As you wire more sub-agents into systems that do things rather than just answer questions, the load-bearing question stops being "is the model smart enough?" and becomes "how do I know the work actually landed?"

An LLM will always tell you what it meant to do. Trust comes from making it prove what it did — a read-back against the source of truth, not a more persuasive prompt.

I'm a Technical Architect building AI-enabled systems. I write about what I actually build — the reliability and design decisions under the hood, not the model hype.

If you've had a sub-agent confidently report work it never finished, I'd like to hear how you caught it. Find me on X @deepakgoyal_ai or LinkedIn.

For years I planned my days around a single number: how much work I could personally get through. Six, maybe eight focused hours, and I'd fill the day to that line. I thought I was planning the work. I was planning me — my hands were the bottleneck, so my capacity was the plan.

That number is now wrong, and staying anchored to it leaves most of the day on the table.

Two kinds of work, and only one is yours

Almost any task is a blend of two kinds of work that behave nothing alike.

Mechanical work — reading files, summarizing, extracting, drafting first cuts, bulk-editing. It has no internal dependencies: item 7 doesn't need item 6. Hand it to a fleet of sub-agents and it happens all at once.

Judgment work — deciding what to do, setting priorities, cutting the dead project, reviewing what came back and saying yes or no. Each call leans on the ones around it and on context only you hold. It can't be handed off, and it can't be parallelized.

For most of knowledge work's history, these two were welded together. You couldn't make the decisions without first doing the reading, so the reading time was simply "the work," bounded by how fast you personally moved through it. Parallel dispatch pulls the two apart. The mechanical half moves to the machine. The judgment half stays exactly where it was — with you, one call at a time.

So your scarce resource was never really time — it was judgment, trapped inside a lot of mechanical hours that hid the difference.

If that sounds like delegation, it isn't — not quite. Delegating to people never exposed this, because a human team has its own execution hours and coordination cost; the mechanical time just moved — it didn't vanish. Collapse that mechanical time toward zero, and your judgment is the only wall left.

The number that sets a day's capacity changed

Once the mechanical half isn't yours, the question that sizes a day flips. It's no longer "how much can I execute?" It's "how much can I decide and review?" — and those numbers are nowhere close.

Here's the concrete shift in my own planning. I used to schedule my days at six to eight hours of work — the amount I could actually do with my own hands. Now I plan them at fifteen to twenty hours of work — measured as what it would have cost me by hand — while spending only a few hours steering it.

Not because I got faster, and not because I started grinding longer. Because most of those hours are mechanical, and mechanical hours aren't mine to grind through anymore. My few hours go to the part that's still mine: deciding what to send off, and judging what comes back — and the judging is real work, not a rubber stamp. Some output is faster to redo than to salvage; catching plausible-but-wrong results is the tax on the leverage.

The plan stopped being a list of what I can do. It became a list of what I can decide — and each decision now pulls far more finished work behind it than it used to.

Why the old estimate quietly wastes the day

If you keep planning against your execution hours, you make the same mistake I did for a while: you under-load the day and never notice.

You block the calendar for the eight hours you can personally work, fill it with eight hours of tasks, and feel fully booked. But the machine that could have carried the mechanical bulk of that work sat idle, because you never scoped for it. You were busy the whole day and still left most of the day's real capacity unused — not because you ran out of hours, but because you ran out of plan.

The fix is to budget the resource that's actually scarce. Estimate the day by your decision-and-review hours, then load work onto it until those hours are full — not until your execution hours are. The mechanical bulk rides along underneath; it isn't what you're pricing.

In practice I scope a day or a sprint at roughly 2–3× what my old execution-bound estimate would have said, then correct from there. (That multiplier is my own calibration from planning-heavy weeks, not a benchmark — yours will track how much of your work actually decomposes into machine-runnable pieces.)

When the split isn't there

This only rewrites your planning when there's a mechanical half to hand off. Often there isn't, and it's worth being honest about where:

• Sequential design — where each step depends on the last decision — is judgment end to end. There's nothing to offload, so your execution hours are still the ceiling.

• A single hard artifact — one proof, one gnarly file, one architecture — has no surface to split.

• An external bottleneck — a slow CI run, a deploy, a human reviewer — doesn't care how many agents you have. The agents just wait in parallel.

For that kind of work, plan the old way: you are the executor, and six-to-eight is honest. The reframe only pays off to the degree your week decomposes — and more of it decomposes than you'd guess once you look for the seam.

Stakes are a different axis, not an exception. Higher stakes don't hand the mechanical work back to you — they make the review half deeper, so you steer less in a day. The split holds; the price of your half goes up.

The skill that actually changes

The instinct says adopting parallel sub-agents means learning to drive a fleet. The prompting is the easy part. The skill that changes is planning itself: seeing the judgment inside your work separately from the machinery around it, and sizing your days by the judgment alone.

Plan your decision-and-review hours, not your execution hours. The day holds as much work as you can steer — which is a great deal more than you can do. And the next time you fill a day to eight hours and call it full, ask what the machine could have been carrying while you did.

I'm a Technical Architect writing about the AI systems I design and how they change the way I work. The capacity numbers above are from planning my own knowledge system; treat them as a starting point to tune, not a law.

It's the most throwaway decision in the system.

Frontier models are converging and commoditizing. The one you pick tends to get matched within a few model cycles, and swapping it out is far less work than rebuilding the task around it. So any edge the model itself gives you is temporary — the same edge everyone else will have a quarter later. The value that actually builds up sits one layer up: in how you redesign the work itself around what AI now makes possible.

I learned this the way most things worth knowing get learned: by spending real effort on the wrong layer first.

The model layer is infrastructure

That last point echoes something Benedict Evans has said about foundational technology cycles: the model layer is this cycle's TCP/IP — genuinely transformative, and not where the durable value gets built. TCP/IP made the web possible, but the value that lasted accrued one layer up — in what the protocol newly allowed, not in the protocol itself.

Same here. GPT-N, Claude-N, Gemini-N are the foundation. "Which model should we use?" is a buying question — it has a cheapest-good-enough answer and a short shelf life. The question that actually shapes an architecture is different: which tasks change shape once AI is embedded? Which steps dissolve, which get re-decomposed, which new ones appear. That question has answers that survive across model generations.

A worked example: the review gate

The clearest case I've worked on is an automated review gate — the kind that checks every incoming change against a house rulebook and flags what's out of line. The first meetings went exactly where I've been describing: which model judges violations best? We treated it as a model-selection problem and spent our attention accordingly.

That framing was the mistake, and the decomposition exposed it. When I broke the rulebook down check by check, the question stopped being "which model" and became "what is each check, actually?" Most of them turned out to be structural questions, not questions of meaning — "does this component reach for something it isn't allowed to?" is a lookup over structure, not a judgment call. Only a minority genuinely turned on intent, the kind of thing — "is this message actually helpful to a human?" — where a language model is the right and only tool for the job.

Here is the part that reframed everything for me. A frontier-model upgrade would have sharpened that semantic minority a little and left the entire structural majority exactly where it was. The bulk of the gate's value — the checks meant to run on every change, every time — never depended on model capability at all. It came from recognizing they were structural and routing them to something that inspects structure. The value was in the decomposition. For most of the system, the model was almost a distraction from the decision that mattered.

I'm deliberately not reprinting the routing mechanics here — that's a cost story I've told elsewhere. The point for this argument is simpler and more lasting: the leverage came from rethinking the task, and a better model would not have delivered it.

Why one layer compounds and the other doesn't

Model gains never last. You upgrade, and within a quarter the whole field upgrades with you; your lead washes out. It's a floor that rises for everyone at once — real, but never yours.

Task decomposition keeps paying off. Once the gate is split into "structural checks" and "genuine judgment," that split doesn't ride on frontier progress. A better model sharpens the judgment slot and leaves the structural checks doing exactly what they did. Each task you re-decompose is an asset that keeps paying off for as long as the task exists — a model swap can't erode it. Each model you pick is a rental that expires.

That's the whole thing in a line: the model is the part that dates fastest; the task redesign is the part that lasts. Your scarce attention belongs on the layer that compounds — which is rarely the one the first meeting wants to argue about.

The hard part is drawing the lines — and you'll draw some wrong

This can read like a truism — of course decomposition matters. The sharper point is that AI raises the stakes on getting it right: the easy path, one prompt for everything, quietly buries the whole structural majority of steps that should never touch a model at all.

And drawing those lines is genuinely hard. The whole approach rests on one skill: telling judgment from structure — which steps truly need a model to understand meaning, and which only look like they do because it's easier to send everything through one prompt.

You will misjudge some. In the gate, a handful of checks looked purely structural but hid a meaning-based edge case that a naive rule would silently miss. The rule I settled on: when a check is ambiguous, send it up to the model rather than let a mechanical shortcut quietly get it wrong. Getting those boundaries right is iterative, and unlike a model swap, every correction you make sticks. The decomposition keeps improving precisely because you keep sharpening where the lines fall.

How to apply it

Next time someone says "let's add AI to X," resist opening with the model. Open with the task:

1. Decompose X into its real steps and decisions — the actual ones, not the category name.

2. For each, ask: judgment or structure? Does a model genuinely earn this step, or does it just look that way because one prompt is easier than one design?

3. Redesign the workflow around that split — deterministic where you can, model where you must — not around the model's interface.

4. Treat model choice as the last and most swappable decision. Pick the cheapest one that clears the bar, and assume you'll replace it within the year.

The instinct is to settle the model question first because it feels like the important one. It's the one that expires soonest. Do the task work first — that's the part that compounds.

I'm a Technical Architect building AI-enabled systems. I write about what I actually build — the decomposition and design decisions under the hood, not the model hype.

If you've re-decomposed a task around AI and found the model choice mattered less than you expected, I'd like to hear where the real leverage landed. Find me on X @deepakgoyal_ai or LinkedIn.

Then it meets production scale, and you find you've been using the most expensive, slowest, least deterministic tool you own to answer questions that were already solved — for free — by something far simpler.

The fix isn't a better model. It's a routing decision made before any work runs: send each decision to the cheapest oracle that can actually decide it — where an "oracle" is just anything that can answer the question, from a regex or a lookup to a small model to the big one. Let me make that concrete with a system I built, then pull out the part that generalizes.

Why "LLM everything" is the default — and why it's wrong

Defaulting every decision to a language model is the natural first move. At prototype scale it's the obvious one. Then production shows you three costs the demo never did:

• Money. You pay per token for decisions a free deterministic function could make.

• Latency. A model call is hundreds of milliseconds to seconds; a structural check is sub-millisecond. Multiply by every decision, every time.

• Non-determinism. The same input can resolve one way now and another way later, because the oracle is probabilistic. For anything that gates a workflow, that's corrosive — people stop trusting a check that flip-flops.

The design treats the expensive oracle as the default. It should be the fallback.

Most decisions have a cheaper oracle that can decide them

Here's the observation that reframes the problem. A large share of the decisions you're tempted to route to an LLM don't require its judgment at all. Their truth condition is something a cheaper oracle can evaluate — a structural check, a lookup, a rule, a smaller model.

The minority that genuinely need the expensive oracle are the ones whose answer depends on meaning or context — the things language models are actually for. The mistake is letting that hard minority set the architecture for the easy majority.

So the design question is never "should we use an LLM here?" It's sharper: for which specific decisions is the LLM the only oracle that can decide? That's a per-decision question, and — crucially — it's answerable upfront, before anything runs.

A worked example: a code-standards enforcement system

The clearest case I lived through was a rule-enforcement system — the kind that scans code against a set of standards and flags violations. The first version routed every rule to an LLM. It demoed well and fell apart the moment it ran on every commit.

Most of the rules I was paying a model to evaluate had a structural truth condition:

• "Does this module import a forbidden package?" — an import-graph lookup.

• "Does every public function have a return type?" — an AST traversal.

• "Does this file match the required naming pattern?" — a regex.

None of those need a model to understand intent; they need a parser to inspect structure. The rules that did need a model were the semantic ones — "is this error message actually helpful?" — where the truth depends on meaning.

So I put a classification step in front of the dispatcher. Before a single check runs, each rule is graded once into two tiers:

The classification is cheap and one-time — a small, fast model is plenty to sort "structural" from "needs-judgment." You pay it once, not on every commit forever. And because that step runs once and gets a human sign-off, its own non-determinism never reaches production: the flaky part is bounded to design time, not runtime. The genuinely ambiguous rules — mostly structural but with a semantic edge — default up to the LLM tier; you'd rather over-power a check than let a deterministic shortcut quietly miss the hard case.

When I ran that split across one rule stream, the majority came back deterministic and only a handful were genuinely LLM-only. Routing accordingly cut the model spend on that stream by roughly 80% — the saving comes from pulling the high-frequency checks (the ones that run on every commit) off the model, not from dropping any coverage. The deterministic checks now run in milliseconds and never flip-flop, so the gate became something people trust.

The part that generalizes

Strip the code-standards specifics and the pattern is domain-agnostic. Anywhere you're tempted to default to the model, the same routing layer applies:

• Classification — a cheap keyword/rule pass first; the LLM only for the ambiguous remainder.

• Extraction — a regex or parser for the structured fields; the LLM only for the free text that resists one.

• Validation — schema and type checks before you ask a model whether something "reads right."

• Escalation — a small, cheap model first; promote to the expensive one only when confidence is low.

In every case the move is the same: grade the decision once, route it to the cheapest oracle that can decide it, and reserve the expensive one for the decisions that genuinely earn it. A one-tier system that sends everything through the same oracle has structurally given up those savings before it starts.

What it costs you

This isn't free. Two tiers means a classification step to maintain and two code paths to keep honest, where a one-tier "LLM everything" system has just one. That overhead is real, and it's why the one-tier version is the right first move: at prototype scale, when you're still discovering what the rules even are, a single model call per decision is the fastest way to learn.

The split earns its keep only when volume shows up — every commit, every item, every time — and the per-decision cost of the model starts compounding. Add the routing layer when the scale is real, not before. Reaching for it on day one is its own kind of over-engineering.

How to apply it

If you're building on an LLM, run one audit:

1. List the actual decisions your system makes — not categories, the real decisions.

2. For each, ask: what's the minimum oracle that can decide this? A rule? A regex? A parser? A lookup? A small model? Or genuine semantic judgment?

3. Route accordingly. Cheap-tier decisions get a real function with a test. Only the judgment calls keep the model.

4. Re-run the audit as you add features. New decisions default to the cheap tier and only graduate to the model when you can name the judgment they require.

The instinct to make everything smart is the expensive one. The discipline is to make each decision exactly as smart as it needs to be — and no smarter.

I'm a Technical Architect building AI-enabled systems. I write about what I actually build — the routing and cost decisions under the hood, not the hype.

If you've split an LLM pipeline into cheap and expensive tiers, I'd like to know where you drew the line. Find me on X @deepakgoyal_ai or LinkedIn.

I'd been running a pipeline that scanned pull requests, generated architectural feedback, and posted comments under my review. The focus was architectural — system design, code structure, standards alignment — not syntax or linting, which automated tools already handle. My assumption going in was that developers would find the comments noisy — too many, too generic, useful only occasionally. AI-generated feedback has a reputation for noise: high volume, generic findings, low actionability on anything beyond linting.

I sent sixteen developers an anonymous survey asking about actionability (1–10), severity tag calibration, which comment categories they valued most, whether they understood the why behind findings, and what they'd change. I expected it to tell me to scale back.

It didn't.

Here's what the data said — and what I changed as a result.

What I expected to find

Low actionability scores. Developers annoyed by AI-generated noise. A signal to reduce volume and tighten focus. That felt like a reasonable prior — and I'd been watching carefully for signs that the reviews were becoming wallpaper.

What I found instead: the pipeline was working better than I thought — with one specific exception I'd been blind to.

The headline numbers

Average actionability: 8.06 out of 10.

More surprising: 94% of developers (15/16) said they understand the why behind comments — not just what was flagged, but the reasoning behind it. One respondent said "sometimes"; everyone else said "always" or "most of the time."

I'd spent significant effort on comment quality — making sure each finding had context, rationale, and a suggested direction. The data said that effort was landing. The reviews weren't noise.

So where was the gap?

The blind spot: severity tags

When I looked at the calibration question, the numbers told a different story than the actionability scores.

Nit was the weakest tag by a wide margin. Only 7 of 16 developers thought it was consistently well-calibrated.

The open responses told me why. Two patterns kept surfacing:

Pattern 1 — Nits with no verdict. Comments tagged [nit] that ended with "okay to ignore if time is tight." Developers picked up on this quickly: if it's truly okay to ignore, don't post it. The comment adds noise without value and erodes confidence in the tag for the comments where it actually matters.

Pattern 2 — Blockers for preferences. One developer gave a precise, constructive example: a review had flagged an architectural approach as a hard blocker and recommended a different direction. Weeks later, a follow-up fix was needed that the original approach would have handled. The feedback: "Some of the strongest comments would work better as 'preferred direction unless runtime proof says otherwise' rather than a hard stop."

That landed. A blocker is a correctness issue, a CI failure, a known production risk. An architectural preference — even a well-reasoned one — is not a blocker until there's evidence it's wrong at runtime. I'd been conflating the two.

The insight: severity tags are a contract, not decoration

In any structured review process, the framing of a finding is a more sensitive credibility lever than its technical correctness.

A developer can tolerate a finding that turns out to be wrong. One wrong finding is one data point. But a mislabeled severity tag contaminates the vocabulary — if [blocker] sometimes means "I really prefer this," developers start discounting all blockers. If [nit] sometimes means "I ran out of things to tag," developers stop reading nits entirely.

Severity tags are a contract with the reader. Every [blocker] must mean the same thing. Every [nit] must be something worth changing, not something to pad the review with.

What developers actually value

The calibration finding was the critical one — but the survey surfaced two more data points worth acting on.

The most-valued comment category was architecture — how the feature is structured and built. 94% selected it.

Safety comments (security, PII, breaking changes) and code clarity (duplication, dead code) both landed around 63%. Tests and CI coverage scored 25%.

That last number was useful. Test commentary scores low not because tests don't matter but because automated tooling (linters, CI bots, static analysis) already covers that surface. Developers don't need another layer of feedback on something that's already caught. The architect review adds value where automated tools don't reach: system design, architectural trade-offs, standards alignment.

Knowing this changed how I allocate review depth. I pull back on test commentary and lean into architectural calls.

The dominant ask: more, not less

The highest-frequency theme in open responses was review cadence. 44% of respondents asked for more frequent reviews, not fewer. Comments ranged from "reducing the turnaround between cycles" to "can we run this several times a day?"

I'd expected the dominant ask to be "less." The actual ask was the opposite.

This matters for how you think about AI-assisted review pipelines. The risk isn't developer fatigue from too many comments — it's developer frustration from feedback arriving too late in the PR cycle to act on without rework. A comment posted 18 hours after a PR opens often arrives after the developer has moved on, which means the cost of incorporating it has gone up.

What I changed

Four concrete changes, directly from the survey:

1. Tightened the nit gate. If the verdict is "okay to ignore," the comment doesn't post. A [nit] must be worth changing to be worth sending.

2. Calibrated the blocker tag. Architectural preferences now read as "preferred direction unless runtime proof says otherwise." [Blocker] is reserved for correctness issues, known production risk, or CI failure.

3. Scaled back test/CI commentary. Where automated tooling (linters, CI bots, static analysis) already covers the surface, the architect review steps back. The value is in what bots can't catch.

4. Added a mid-day review slot. The 44% frequency signal was the loudest ask in the survey. The pipeline now runs at multiple points during the working day rather than once, reducing the gap between a PR opening and the first round of feedback.

None of these are dramatic changes. Each one came directly from a developer telling me, with data behind it, where the signal was breaking down.

The operational test

If you run structured reviews — AI-assisted or otherwise — read the last ten comments you posted and ask: do the severity tags mean what the reader thinks they mean?

If [blocker] has appeared on an architectural preference, you've inflated it. If [nit] has appeared on something you'd genuinely accept unchanged, you've diluted it. The cost isn't measured per-comment. It compounds across the corpus.

Survey your team. Ask specifically about tag calibration, not just overall satisfaction. The headline scores will probably be better than you expect. The calibration scores will tell you where the actual work is.

I run AI-assisted code reviews as part of a Technical Architect role on an engineering team spanning backend, frontend, mobile, and QA. The pipeline generates architectural findings; I review and post them. After six weeks, I ran a 16-response anonymous survey to validate assumptions about how the practice was landing. The data changed how I operate.

If you've surveyed your team on AI-assisted reviews — what surprised you most?

That's not a capability failure. It's family bias: a structurally correlated blind spot that no amount of prompting fixes, because the judge and producer share training patterns — they find each other's outputs "familiar" in a way that operates below the prompt layer. The 2025 research on this is now strong enough to act on.

I built a two-judge review loop — one Anthropic model, one OpenAI model — with a convergence rule that defines what "agreement" actually means. After ~75 review items and dozens of iteration loops, the pattern held: the second judge caught a class of errors the first couldn't, not because it was smarter, but because it wasn't correlated with the producer.

There's also an open-source tool so you can run it against your own content.

Why single-judge fails quietly

Frontier models are competent judges. That's precisely why this failure mode is dangerous — the verdicts look authoritative.

Spiliopoulou et al.'s "Play Favorites" (Aug 2025) showed that GPT-4o and Claude 3.5 Sonnet don't just over-rate their own outputs — they over-rate outputs from their entire model lineage. Wataoka et al. (arXiv 2410.21819, rev. Jun 2025) traced the mechanism to pattern familiarity: judges find text from familiar architectures easier to process and rate it higher. Adaline's April 2026 analysis found frontier judges failing more than 50% of bias stress tests, with the practical takeaway: "Separate your judge's model family from your generator's."

The solution isn't a smarter judge. It's a judge that can't share the producer's blind spots.

The convergence rule

I use two independent judges per item:

• Judge A: Anthropic (Claude Sonnet by default)

• Judge B: OpenAI (GPT-4o by default)

Each sees the same content and criteria. Neither sees the other's output. Each returns:

• confidence — certainty in the verdict, 0–100

• verdict — APPROVED, FLAGGED, or UNCERTAIN

• rationale — one sentence

A decision is trusted when two conditions both hold:

Both are required. Average alone misses divergence; spread alone misses low-confidence agreement.

Examples from actual runs:

My calibration: avg ≥ 75, spread ≤ 15, max 4 iterations. Tune these against your corpus — the shape transfers, the numbers don't.

When a verdict doesn't meet both conditions, I don't discard the item — I escalate.

The escalation ladder

The escalation trigger is specific: if improvement between iterations is ≤ 5 confidence points, more of the same model won't close the gap.

Two things I learned running this:

I learned not to escalate both judges when one was diverging. If Judge A was consistent across iterations and Judge B was all over the place, escalating both burned budget without isolating the signal. The ladder reflects this — it steps up Judge A (Anthropic) first, since in my experience Opus is more reliable than Sonnet on genuinely ambiguous items, while Judge B stays on gpt-4o through L3.

I capped at 4 iterations. After 4 loops without convergence, the item is genuinely ambiguous. The right response is a human decision, not a fifth model call.

See it running

uv run python3 validate.py \
  --content "Cache relies on TTL alone, no explicit eviction path." \
  --criteria "Flag any caching pattern without an eviction or fallback path."

Dual-LLM Validator
  Judge A : claude-sonnet-4-6
  Judge B : gpt-4o
  Rule    : avg >= 75  AND  spread <= 15  |  max 4 iters

Iteration 1 / 4
  Judge A  claude-sonnet-4-6   confidence  95  FLAGGED
  Judge B  gpt-4o              confidence  95  FLAGGED

  avg 95.0  |  spread 0  |  CONVERGED ✅

Result     : FLAGGED
Iterations : 1

Rationales:
  Judge A: TTL-only caching creates silent stale reads
           when upstream data changes without eviction.
  Judge B: No eviction path means stale entries persist
           until TTL expires, regardless of data changes.

When it doesn't converge on the first pass:

Iteration 1 / 4
  Judge A  claude-sonnet-4-6   confidence  85  FLAGGED
  Judge B  gpt-4o              confidence  48  UNCERTAIN

  avg 66.5  |  spread 37  |  NOT CONVERGED (verdicts disagree)

  >> Trust delta stalled — escalating to L2

Iteration 2 / 4  [L2]
  Judge A  claude-opus-4-7     confidence  80  FLAGGED
  Judge B  gpt-4o              confidence  76  FLAGGED

  avg 78.0  |  spread 4  |  CONVERGED ✅

L2 escalates Judge A to Opus (stronger on ambiguous items); Judge B stays on gpt-4o — the divergence was on the Anthropic side.

Try it yourself

git clone https://github.com/deepakgoyal-ai/dual-llm-validator
cd dual-llm-validator
uv sync
cp .env.example .env   # then open .env and fill in your keys

uv run python3 validate.py \
  --content "your content here" \
  --criteria "your review criteria here"

Custom models and thresholds:

uv run python3 validate.py \
  --content-file review.txt \
  --criteria-file criteria.txt \
  --model-a claude-opus-4-7 \
  --model-b gpt-4-turbo \
  --avg-floor 80 \
  --spread-max 10

Exit code 0 = converged. Exit code 2 = human review required. Pipe-friendly.

The operational discipline that matters most

Two things that aren't about the models at all:

I re-validate after every fix. First dispatch + fix applied ≠ done. A verdict is closed only when a re-validation pass converges on the corrected content. Skipping re-validation means I've applied a fix that passed one judge once — which is the original failure mode.

I managed two token budgets, not one. Running two providers concurrently means two rate limits refilling on different schedules. I alternated between providers across dispatch batches rather than parallelizing everything. Parallel looks faster until one provider hits a rate limit 40% through and stalls everything.

What I'm still unsure about

Do these thresholds transfer? I calibrated avg ≥ 75 and spread ≤ 15 against one corpus. There's no reason to expect them to be correct for a different review task. Calibrate against your own items before trusting the defaults.

Did I trade family bias for a different correlated bias? Cross-family reduces family bias — the 2025 papers document this. It doesn't eliminate correlated failure. Two models trained heavily on the same internet text could still share blind spots I haven't characterized.

When shouldn't you do this? If the cost of a wrong verdict is low, single-judge is cheaper and probably fine. If you have labelled data and bandwidth, a fine-tuned single judge likely outperforms two general-purpose ones. This pattern is the 0-to-1 case — no training data, real stakes, need something better than one model family's opinion.

The short version: two judges, different families, convergence = avg ≥ floor AND spread ≤ max, cap at 4 iterations, accept ambiguity rather than escalating forever.

The rest is calibration.

I'm a Technical Architect building AI-enabled systems. Writing about what I actually build — not what I think I should build.

If you're running a review pipeline and have calibrated your own thresholds, I'd like to know what you landed on.

1. The cracks: five weeks in

Five weeks into using the AI second brain I'd built, three things were nagging at me.

The skills were getting fat. The project list was lying to me. Every area had its own dialect.

None of them were bugs. All of them were structural.

Quick context if you missed the original post: SMRITI v1.0 is an open-source second brain that pairs Obsidian with Claude Code. 13 markdown skill files give Claude Code daily-companion behaviors — morning brief, capture, weekly review, knowledge extraction. It worked. The Galaxy crossed 100 atomic notes. The morning brief landed on time.

By week four, those cracks were impossible to ignore. v2.0 is the rebuild that fixes all three. Below is what hurt, what I changed, and the cross-cutting lesson I didn't see until the rebuild was done.

2. One command, doing everything

The pain

One evening I was debugging why /close-my-day had skipped a step. I opened the skill file. Scrolled. Scrolled more. Skill file: 127 lines.

/close-my-day was doing everything. Pulling Todoist completions. Reconciling daily notes. Running the weekly review on Sundays. Extracting Galaxy candidates. Drafting tomorrow's quests. Mini-triaging captures.

Each was a block of inline logic. Token cost climbed every time I invoked it. Changing one phase meant reading all the others.

13 skills, most of the big ones bloating the same way. The biggest one ran past 200 lines.

The fix: skill = touchpoint, agent = depth

The skill stays thin. It's what you type. The depth moves into specialized agents that the skill dispatches.

/close-my-day  (79 lines — was 127)
  │
  ├─→  Task Sync agent           (pull Todoist completions)
  ├─→  Auto-Link agent           (Friday cascade)
  ├─→  Weekly Review agent       (Sunday cascade)
  └─→  Galaxy Extractor agent    (pull atomic candidates)

Mini-triage and tomorrow's quests draft stay inline in the skill — they're light enough.

• Skills (6). /start-my-day, /close-my-day, /checkpoint, /braindump, /triage, /consolidate. Each is a thin coordinator — read state, dispatch, present results, write back.
• Agents (8). Specialized sub-capabilities the skills dispatch. The clearest example is the Task Sync agent: it pulls completed Todoist tasks, finds the matching tids in vault Project files, ticks the checkboxes, and writes Completed:: fields. 270 lines of focused logic. Yes, that's longer than v1.0's biggest skill — that's the point. Depth lives in agents now. The skill stays thin; the agent takes the weight.

Why it matters

Each agent picks its own model. Galaxy Extractor runs on Opus because synthesis quality matters. Inbox Triage runs on Sonnet because it's template-driven. Task Sync runs on Sonnet because it's transactional. A monolithic skill can't optimize per-phase. A dispatcher can.

Agents run in parallel. Friday close fires Auto-Link and Librarian in one message. Wall-clock drops to the slowest agent, not the sum.

Skills shrank. /close-my-day went from 127 → 79 lines (38%). /start-my-day went from 148 → 107 (28%). The hard logic still exists — it just lives in agent files where it belongs.

Net: 13 skills → 6 skills + 8 agents. Fewer commands. More capability.

3. The list that always lied

The pain

Every Friday I'd open the area's Initiatives.md to plan the next sprint. Table of projects — status, priority, effort, owner. Looks great in screenshots.

It was always wrong. I'd close a project. Forget to update the table. Next Friday, three rows said "active" for projects I'd shipped two weeks ago.

The Project files had the truth. The companion file had a snapshot nobody re-snapped.

And every project lived in one flat 01 - PROJECTS/ folder. Work next to learning next to personal. The folder lost its signal.

The fix

Two changes, one outcome.

Kill the companion file. The Initiative Index is now a live Dataview query (Dataview is an Obsidian plugin that reads frontmatter from notes and renders queryable tables). It reads directly from the Project files themselves. No more hand-maintained tables.

Add this to the area's Playbook under ## Initiative Index:

TABLE status, priority, effort, planned, completed
FROM "01 - PROJECTS/Work"
WHERE status != "cancelled"
SORT priority ASC, planned ASC

The Playbook shows the truth because the truth is the Project files.

Move Projects under per-area folders. Instead of a flat folder, Projects live under per-area subfolders:

01 - PROJECTS/
├── Work/
│   ├── 001-q3-platform-migration.md
│   └── 002-pr-policy.md
├── Learning/
│   └── 014-fastapi-deep-dive.md
└── Personal/
    └── 003-q3-personal-checkpoint.md

Project IDs are area-scoped — work-001, learning-014. Never reused. When you cancel a project, the file stays in place with status: cancelled. The ID stays bound to the file. No tombstoning, no ambiguity, no "which work-014 are we talking about?"

The locked schema

Each Project file carries 18 frontmatter fields — status, priority, effort, planned/completed dates, recurrence pattern, Todoist linkage, tags. Each field earned its place by being load-bearing for dispatch, queries, or stability.

The point isn't the field count. The point is that the schema forces decisions to be visible.

• status is one of 8 values. No freeform. You can't fudge a project as "kinda done."
• effort is one of 7 values. You can't claim a 30-minute task is "ish, maybe."
• recurring: true + iteration: N lets one file hold 50 iterations of weekly work instead of generating 50 files.

Feels restrictive for a week. Then it's muscle memory.

4. Five areas, five dialects

The pain

I move between areas every day. Work in the morning. Learning at lunch. Personal in the evening.

Each area's CLAUDE.md had drifted into its own vocabulary.

Work documented its weekly rhythm as "Operating Loop." Learning called it "Daily Cadence." Personal didn't document it at all.

When I wanted to know the rhythm, I had to translate every time. So did the AI. Every skill reading an area's CLAUDE.md had to handle three conventions.

The fix: every area gets the same Playbook

Six canonical sections, always the same, always in the same order. Here's a real Learning-area Playbook filled in:

## Mission
Learning is where I spend protected hours on AI/ML mastery — concept
clinics, hands-on experiments, and Galaxy notes that compound into
long-term competence. This is the area I won't let work crowd out.

## Operating Rhythm
### Daily
- 12:00–14:00 protected study block (Slack DND, no calls)
- 21:00 close ritual + Galaxy candidates pass

### Weekly
- Sunday: next topic plan + audit current topic for AI-moves-fast drift
- Galaxy target: 2 atomic notes / week

## Initiative Index
(Dataview query renders open + closed Projects from
 01 - PROJECTS/Learning/)

## Library
- Phase plan, topic prerequisites, Galaxy MOC entries
- Reference papers, cohort notes, external course bookmarks

## Signal Log
- Topic-level surprises ("retrievers + rerankers asymmetric" — 2026-05-20)
- Blog post candidates surfaced mid-study
- Tools to investigate (LangSmith, Phoenix evaluation)

## Cross-cuts
- Inputs: research papers (RSS / Readwise) → Inbox → triage → Learning
- Outputs: Galaxy notes → blog posts; Decisions → Work architecture

When every area has the same shape, three things change.

You move between areas without translation. "What's open in Learning?" → Initiative Index. "What's the rhythm?" → Operating Rhythm. No mental tax.

AI doesn't relearn. Every skill knows where the Initiative Index lives. The Auto-Link agent knows Library entries are link candidates. The Inbox Triage agent knows every area has a Signal Log where a half-baked idea can sit until it ripens.

New areas are cheap. setup.sh ships an interactive Area loop — type a name, it copies the Example-Area template, substitutes your name across the Playbook + CLAUDE.md, creates the matching Projects folder. ~30 seconds per area.

The trade-off: some loss of structural freedom. I wanted a "Reading Queue" section in my Learning area's Playbook — it ended up inside Library. Library is for stable reference material; a Reading Queue is more of a status surface. The fit isn't perfect, but it's good enough. That's the price of conformity. I'll take it.

5. Why Standards matter more when AI is your editor

The cross-cutting lesson I didn't see until v2.0 was done.

v1.0 had no Standards layer. Every skill was its own interpretation of how the vault should work. Every area's CLAUDE.md was a dialect. Drift was automatic — and worse, drift was invisible because everything still worked.

And here's why this matters when AI is the primary editor.

Every skill is an instruction file. When you author skill N, you're consciously matching skill N-1's conventions. When you author skill N+3 three weeks later, you've forgotten which conventions were deliberate and which were accidental. Two incompatible versions of the same idea both work. Both ship. Drift accumulates.

On a team, PR review catches the drift. Solo with AI, there's no PR review.

The Standard is the enforcement layer. It's the thing that lets you author skill N+3 and have it land coherent with skills 1 through N+2 because there's a written reference for "how we do it here."

v2.0 ships 8 canonical Standards under 02 - AREAS/SADHANA/Standards/. The four highest-leverage ones:

• Project Template — Epic / Task / Sub Task hierarchy, the 18-field schema, status vocabulary.
• Initiative Index Standard — the Dataview query shape, status grouping, column order. This is what makes Section 3's queryable project structure possible.
• Frontmatter Standards — the 5-field minimum on every note. Got muscle-memory in three days.
• Skill Authoring Standard — skill size limit (110 lines hard cap), frontmatter, the pre-add gate (more on this below).

The other four cover tags, themes (optional), permissions, and model routing.

The pre-add gate

The single most useful artifact across all 8 Standards is a three-question gate in the Skill Authoring Standard. Before adding skill N+1:

1. Can an existing skill extend to cover this?
2. Can a hook, cron, or agent replace the need for a user-invocable command?
3. If neither — retire one skill, or justify with a concrete re-eval trigger.

v1.0 hit 13 skills because there was no gate. v2.0 sits at 6 because the gate fires on every new idea. The cap counts user-invocable skills only — agents and standards don't count, so the system grows in depth without sprawling at the surface.

6. What I didn't see coming

Writing a new skill feels easier now. The hard work happens in the agent. The skill is just a switchboard. I expected the split to add cognitive overhead. It removed it.

Standards stuck faster than I expected. Once Frontmatter Standards locked the 5-field minimum, every note from that day forward had those 5 fields without me thinking. About three days of deliberate practice, then muscle memory.

L1 cache earned its keep. L1 is a pre-assembled context file at .claude/cache/L1-static.md — tags, frontmatter rules, sync rules, area context, all in one read. Replaces 5-10 source-file reads per skill invocation. I haven't measured token counts precisely (the prompt size varies too much for one number), but the bigger win was perceived latency. Skills feel snappier.

Standards made new agents cheap to write. Inbox Triage was the last agent I added during the rebuild. I opened the Skill Authoring Standard, copied the structure, filled in the depth — done in an evening. Without the Standard, this would have been three evenings of "what's the right pattern here?" The retrofit pain I paid in v2.0 buys me speed in everything that comes after.

7. What I'd do differently

Start with Standards, not skills. v1.0 → v2.0 was a forced migration. If I'd written even a minimal Standards layer first and authored skills against it, the rebuild wouldn't have been necessary.

The recursive Standard problem. How do you write a Skill Authoring Standard with no skills to test it against? I dodged this in v1.0 — wrote zero standards, shipped 13 skills, then tried to retrofit Standards on top. Half the rules in v2.0's Skill Authoring Standard are reverse-engineered from v1.0 mistakes. The right move is iterative: write a minimal Standard, ship one or two skills against it, refine the Standard, ship more.

Documentation outpaced the code. I assumed 8 Standards docs would take a weekend. They took five evenings. The Standards layer ended up roughly 3× the line count of the skills it governs (~1,500 lines vs ~460). If you're building one of these, budget the writing time as its own phase.

Agents are easier to retire than skills. I planned 10 agents. Shipped 8. Two got cut during integration testing — not dispatched by any shipping skill, depended on infra I hadn't built. Cutting an agent meant deleting one file. Cutting a v1.0-style skill would have meant breaking a user-facing command and forcing every user (me included) to relearn the flow. Agents are cheap to add and cheap to remove. Skills aren't.

8. What v2.0 doesn't ship

I want to be transparent about what v1.0 features got cut or moved:

• Inline capture (/capture in v1.0) — folded into /braindump. One capture surface, less to remember.
• Standalone connect skill (/connect) — became the Auto-Link agent, dispatched automatically from Friday and Sunday /close-my-day. Removes the manual command.
• Standalone weekly review (/weekly-review) — became the Weekly Review agent, auto-dispatched from Sunday /close-my-day.
• Standalone extract / promote skills — became the Galaxy Extractor and Knowledge Promoter agents, dispatched by /consolidate and the weekly cascade.

If you used any of these as standalone commands daily in v1.0, the muscle memory shifts in v2.0 — the work still happens, but it's auto-dispatched rather than typed.

9. Try it

v2.0 is open source. Three steps:

git clone https://github.com/deepakgoyal-ai/smriti.git
cd smriti
./setup.sh

setup.sh is interactive — 8 steps. Name, day-start, day-end, optional Todoist Sprint Board IDs, then the Area loop where you scaffold your first real Areas. ~5 minutes start to finish on a fresh clone.

Migrating from v1.0? SETUP.md has a dedicated section. Breaking changes are real — vault layout shifted, 7 skills converted to agents, Standards became canonical. My own vault migration took about 30 minutes (file moves + frontmatter additions, no MCP config changes).

What you need:

• Obsidian (free)
• Claude Code (Pro plan recommended)
• A clean clone or a v1.0 vault to migrate
• A daily journaling habit — if you don't already journal daily, this won't compound

Daily time commitment: ~5 minutes morning (/start-my-day), captures during the day (zero overhead — just type), and 5-10 minutes evening (/close-my-day). The Sunday close is the longest at ~15 minutes because it runs the weekly review.

What ships: 6 skills, 8 agents, 8 Standards, 1 L1 cache template, opt-in Todoist sync, an Example-Area scaffold, a sample Project, a couple of sample Galaxy notes.

If v1.0 still works for you, stay. v2.0 is what I'd ship if starting from scratch today.

Notes, skills, agents, standards — all markdown. Your knowledge is yours.

Built by Deepak Goyal. Follow the journey: GitHub · LinkedIn

1. The Problem

My notes were in Obsidian. My tasks were in Todoist. My AI remembered neither.

I'm a Technical Architect. Decisions, meeting notes, architecture reviews, a long-term learning goal — all producing knowledge daily. I had a decent PKM setup. But the AI layer was disconnected. Every Claude conversation started from zero. Every session, I re-explained my vault structure, my workflows, my preferences.

The real problem isn't note-taking. It's that your AI forgets everything the moment you close the terminal.

I wanted an AI that knows my vault, runs workflows when I say /start-my-day, tracks what's overdue, and builds a knowledge graph alongside me. Not a chatbot. A companion that compounds.

2. The Idea and the Four Pillars

I didn't invent this from scratch. The spark came from Karpathy's LLM Wiki — using LLMs to build and maintain personal knowledge bases in plain markdown. Not RAG. Not vector databases. Just an AI that reads your files and writes back to them.

The community ran with it. People built Obsidian + Claude Code implementations, wiki compilers, auto-linkers. What I saw was a deeper pattern: separate storage from intelligence.

• Obsidian = storage. Plain markdown. No vendor lock-in. You own it forever.

• Claude Code = intelligence. Reads files, runs skills, creates notes, remembers context.

• MCP = the bridge. Claude Code reads and writes to the vault directly.

What's original in my approach: the skill system (13 markdown instruction files, not code), the four-pillar philosophy, the Architect's Codex vocabulary, and the daily orchestration flow.

The four pillars — all Sanskrit names — each own one responsibility:

SMRITI (Memory) — The vault. Plain markdown, owned forever. Atomic knowledge that compounds.

SARATHI (Charioteer) — Claude Code as the AI companion. Like Krishna guiding Arjuna's chariot — it sees the battlefield and executes the strategy. 13 skills are its capabilities.

SADHANA (Practice) — The operating principles. Partnership model, autonomy levels, daily rhythm. The constitution that governs how the system operates.

SIDDHI (Achievement) — Long-term goal pursuit. Structured weekly plans, session notes, progress tracking. Any ambitious multi-week goal.

They reinforce each other: SMRITI stores what you learn. SARATHI captures and connects it. SADHANA keeps it consistent. SIDDHI gives it direction.

If Claude Code disappears tomorrow, I keep every note.

3. The Architecture

YOU (Ideas, Decisions, Judgment)
  |
  v
CLAUDE CODE (Reads vault → Runs skills → Writes back)
  |
  +---> Obsidian MCP (vault read/write)
  +---> Todoist MCP (task sync, optional)
  |
  v
OBSIDIAN VAULT (Plain markdown — you own this forever)

The vault follows IPARAG (Inbox, Projects, Areas, Resources, Archives, Galaxy) — six folders that route every piece of information:

00 - INBOX/          Capture everything here first
01 - PROJECTS/       Active work with end dates
02 - AREAS/          Ongoing responsibilities (no end date)
03 - RESOURCES/      Reference material
04 - ARCHIVES/       Done or inactive
05 - GALAXY/         Permanent atomic knowledge

The Galaxy is the crown jewel. Every note is one atomic idea — a claim stated as a title, written in 3-5 sentences, linked to related ideas. A vault with 1000 unlinked notes is a graveyard. A vault with 100 well-connected Galaxy notes is a thinking tool.

But the folder structure is just the skeleton. Four layers make it intelligent:

The whole system is ~2,500 lines of markdown instructions and templates. Built over 3 weeks of evenings.

4. A Day in the System

Morning: /start-my-day

One command. Claude reads state files, counts Galaxy notes, checks overdue workflows, pulls Todoist tasks, generates a brief:

> MORNING BRIEF — Friday, April 25
> Focus: SIDDHI W3 Day 5 + Sprint close
> Atlas: 31 · Drop Zone: 0
>
> SIDDHI: Week 3 — 14.5/15 hrs | 22/29 tasks | on track
>
> Active Quests:
> 1. SIDDHI study session — Hybrid Search
> 2. PR Review scan #2
>
> Bounties:
> 3. Update property IDs (15m)
> 4. LinkedIn post draft (30m)

Creates the daily note, fills the session log, sets up quests (big tasks) and bounties (small jobs). Most important task goes first.

During the day: /capture

Mid-meeting thought: /capture the retry logic on event grid should use exponential backoff with jitter. Claude detects the domain, structures a note, suggests wikilinks, offers to create a Todoist task. Captured in seconds.

Need to close the terminal mid-task? /checkpoint saves full context — plan, progress, next step. Resume later with "continue."

Evening: /close-my-day

Claude reads my daily note, harvests uncaptured action items, syncs completed Todoist tasks, generates a summary:

> SESSION CLOSED — Friday, April 25
> Quests: 2/2 · Bounties: 3/4
> Galaxy candidates: 2 — run /extract
> Tomorrow: SIDDHI Day 6 + Blog draft

Drafts tomorrow's quests, rates intention score, and on Fridays generates a weekly summary. Under 2 minutes.

5. The Skill System and Knowledge Pipeline

No code. Zero.

Each skill is a markdown file — plain text instructions that tell Claude Code what to do. Here's /extract, simplified:

/extract — Knowledge Extraction to Galaxy

1. Read & Identify Atomic Ideas
   Read source. Find ideas that pass:
   - Assertion test — statable as a claim?
   - Standalone test — readable without source context?

2. Check Duplicates
   Search Galaxy for existing notes on same concept.

3. Draft Galaxy Notes
   Assertion-style title, 3-5 sentences, 2+ wikilinks.

4. Present for Approval
   Show previews. User approves/modifies/rejects each.

5. Create Notes
   On approval: create in Galaxy, update source.

No API calls. No function definitions. No deployment. Claude reads this file when you type /extract and writes results back to your vault.

The 13 skills form a knowledge lifecycle:

The pipeline moves knowledge through stages:

• Capture → raw thoughts land in inbox

• Triage → route to the right folder

• Extract → pull atomic ideas into the Galaxy

• Promote → synthesize mature knowledge

• Connect → build the graph

Because skills are markdown, you can edit them in 30 seconds. My morning brief was too long — I added "Keep output under 400 words" to the rules. Wanted quest/bounty separation — added a template to the brief format. The system evolves at the speed of editing a text file.

6. What Surprised Me — and What Breaks

The AI actually follows instructions. I expected drift, hallucinated steps, constant correction. Instead, Claude follows markdown instructions with surprising fidelity. "Ask before creating" — it asks. "Skip Todoist if not configured" — it skips. Markdown-as-code works.

The system self-improves. Claude Code has auto-memory — it saves your corrections and preferences across sessions. I said "stop summarizing what you just did." It saved that as feedback and never did it again. The vault feeds the AI. The AI feeds the vault. Flywheel.

Plain text wins. Notion databases, Roam graphs, Tana supertags — I've tried them. They all hit the same wall: AI can't access it, data is locked, or the tool pivots. Markdown in a folder will outlive every app. Swap the intelligence layer anytime. The knowledge layer is permanent.

What breaks

Context window limits. A skill that reads too many files in one session burns through context. I hit this with /weekly-review trying to read 7 daily notes at once. The fix: L1 cache pre-assembles common context so skills don't re-read source files.

Permission prompts. Claude Code asks for approval on every file write by default. Useful for safety, annoying at scale. You need to configure allow-rules in settings.json for your vault paths — this takes trial and error.

The AI occasionally over-helps. Tell it to update a date in a state file and it might also "improve" surrounding content. The fix: explicit rules in skills — "touch only what's needed, match existing style." It works, but you learn this through corrections.

Todoist sync is fragile. Task IDs, project IDs, section IDs — one wrong ID and tasks land in the wrong project. I burned an hour debugging this on day one. The setup script handles it now, but it's the least forgiving part of the system.

7. What I'd Do Differently

I built 13 skills before locking in the daily habit. Wrong order. Should have started with 3 — /start-my-day, /close-my-day, /capture — and lived with them for a week before adding the pipeline skills. The knowledge pipeline only matters once you have enough notes to pipeline.

I optimized tokens before the flow worked. Built an L1 cache to reduce context usage before I had the basic skills running cleanly. Cache optimization matters at scale. At the start, just get the flow right.

The one thing I got right: anchoring to a physical trigger. /start-my-day follows making warm water after my morning walk. Physical trigger, not digital. The system followed the habit, not the other way around.

8. Get Started

SMRITI is open source. Three steps:

git clone https://github.com/deepakgoyal-ai/smriti.git
cd smriti
claude
# Paste: "Set up SMRITI for me. Read SETUP.md and follow the instructions."

Claude auto-detects your system, configures MCP, walks you through setup. Open the folder in Obsidian as your viewing layer. Done.

What you need:

• Obsidian (free)

• Claude Code (Pro plan recommended)

• 30 minutes (longer if you hit MCP auth issues — see the repo README)

What you get:

• 13 skills for daily orchestration, knowledge management, and goal tracking

• IPARAG vault structure that routes every piece of information

• A knowledge pipeline that compounds over time

• An AI companion that remembers your context across sessions

Ships with sample notes, templates, and a full operating system you can customize.

Your notes are markdown. Your skills are markdown. Your knowledge is yours.

Built by Deepak Goyal. Follow the journey: GitHub · LinkedIn