Search for a command to run...
I'm building an open-source system to interrogate the AI-research firehose — RAG now, a multi-agent researcher by the end, in public over five parts. Part 1 is the foundation: the architecture, the ingestion, and the eval bug that would have shipped meaningless numbers.
Couldn't agree more with this! Often we overcomplicate things, but keeping it structured like this solves half the problem. Thanks for putting this together.
A sub-agent that writes to your system will tell you it succeeded. It's reporting what it intended to do, not what actually landed — and the gap is where silent data corruption lives.
You can plan a day around what you can do, or around what you can decide. Until recently those were the same plan, so it never mattered which you meant. Parallel sub-agents pull them apart — and the i
When a team sets out to add AI to something, the first real argument is usually about the model. Which provider, which version, this one or the one shipping next month. It feels like the decision ever
When you build on top of a language model, routing every decision to the model is the path of least resistance. One interface, one prompt, one place to add logic. It works beautifully in a demo. Then
Deepak Goyal
9 posts
Architecture decisions, not opinions. AI systems, system design, and what actually works in production.
Nobody can keep up with AI research anymore. On ArXiv — the open-access preprint server where nearly all AI/ML work lands first — the three machine-learning categories I care about (cs.AI, cs.CL, and cs.IR) publish more papers in a week than anyone can read in a month. The tooling for interrogating that firehose — "what do the last hundred papers say about reranking, and can I trust the answer?" — barely exists outside closed products.
So I'm building one, in the open. A system that ingests a pinned corpus of ArXiv papers and answers real questions against it: retrieval-augmented generation (RAG) today, a multi-agent researcher-critic-synthesizer by the end. The code is MIT-licensed on GitHub, the write-ups live at deepakgoyal.ai, and it's reproducible down to the exact paper. It ships over five parts — each a working release and a post like this one. Not a tutorial, and not a learning diary: an architect building a real system and showing the decisions.
This post is Part 1: the foundation — the corpus, the ingestion pipeline, the retrieval architecture, and the first evaluation framework. It's the least glamorous slice and the one everything else stands on.
Here's the arc, so you know what you're reading toward:
And one rule runs through all five, the whole reason I write these the way I do: measurement before claim. Every comparison I publish shows its methodology, its sample size, its cost, and a command you can run to reproduce it. No "it's better" without the eval that says so.
Which is exactly why Part 1 has to open with a confession: my first eval was measuring nothing.
I built a retrieval evaluation, ran it, and got a clean table of numbers. Then I looked closer and realized the metric was scoring itself — the way I'd defined "relevant" was rigged, by construction, to make retrieval look good. I was one commit from publishing a table that looked rigorous and meant nothing. (The exact mechanism is in the retrieval section below, where the numbers live.)
I cut the broken metrics and kept only what survives scrutiny. That's what this post is really about: the interesting part of building RAG isn't the pipeline, which is largely solved — it's knowing whether the thing works, and being honest when your own measurement can't tell you.
The rest is the foundation, in the order I'd build it again: the corpus, the pipeline, and the two evals — one I threw out, one I'll stand behind.
ArXiv because the corpus is stable (papers don't change under you), acquisition is trivial, the license is friendly, and the content matches the audience — AI/ML research in cs.AI, cs.CL, cs.IR.
100 papers because it's large enough to escape toy-corpus saturation — where every query trivially retrieves the same handful of chunks — without burning a rate-limit budget on volume I couldn't yet justify. The corpus is pinned: 100 papers, 11,187 chunks. I'll scale to a few hundred only when a gold-labelled eval shows a recall ceiling worth chasing, not before.
Free-first because most of this workload doesn't need a paid model. BGE-small embeddings run locally, the ChromaDB vector store is local, the cross-encoder reranker is local, and the default answer model is a free Groq-hosted Llama-3.3-70B. Paid models come out only for documented A/B comparisons on small samples. Total project budget: ~$15.
That's a decision, not a limitation. A RAG pipeline that only works when every call hits a frontier API isn't a pipeline — it's a bill.
The whole thing is one line of dataflow:
ingest → chunk + embed → ChromaDB → retrieve → (rerank / fuse) → generate → cite
exposed over FastAPI (/search, /ask, /chat, /ingest, /health).
Ingestion is the unglamorous part, and it's where trust is won or lost. It hits the ArXiv API with a 3-second politeness delay and automatic retries, and it's idempotent per paper — an interrupted run resumes instead of restarting. Any PDF that won't parse is logged in the run summary and skipped, never silently dropped or left to crash the batch. The output is data/manifest.jsonl — a committed record that pins every paper by its ArXiv ID. That file is the corpus. Anyone can rebuild the identical 100 papers from it.
Those first two items aren't decoration: ArXiv rate-limits aggressive clients, and an early impatient run earned a batch of 429s and 503s and downloaded nothing. Slowing down and checkpointing fixed it — the pinned rebuild then pulled all 100 papers with zero parse failures.
Retrieval is one idiom, repeated. Every retriever is the same shape — a retrieve(query, top_k) -> list[dict] closure over one consistent chunk dictionary — so a new strategy slots in without touching the callers. Six ship in Part 1, and since these names recur in the results below, here's each in a line:
One design note I'd repeat on any RAG build: implement fusion once. A single Reciprocal Rank Fusion (RRF) routine — rank each list, score every item by 1 / (60 + rank) (60 is RRF's smoothing constant, unrelated to the top-k cutoff), sum across lists — serves both hybrid retrieval (fuse 2 lists: dense + BM25) and multi-query (fuse N lists: one per variant). They look like different features; they're the same operation. Writing it twice is how the two copies drift.
The conversational path is history → LLM rewrite → retrieve → answer, with a bypass_rewrite escape hatch for the failure mode where the rewrite over-constrains a follow-up and retrieval returns nothing.
Here's where most RAG write-ups quietly cheat. Two evals ran on this corpus. Only one produces a number I'll stand behind — and before either table, here's exactly how each number is computed, because a metric you can't reproduce is just a vibe with a decimal point.
Retrieval — category-precision@k. Per question: take the top k retrieved chunks (k=5), look up the source paper of each, and check whether that paper's primary ArXiv category is one the question expects (its source_hint, e.g. cs.IR cs.CL). The score is the fraction of the five that match; the reported figure averages that over every question with a usable hint (27 of the 33). It answers one narrow question — "did retrieval stay in the right topic?" — and nothing more.
Generation — claim-level faithfulness. Faithfulness measures how much of an answer is grounded in the retrieved context. A fixed judge model scores it in two steps, at temperature 0:
supported, unsupported, or not_found.The question's score is supported ÷ total claims. So if an answer makes four claims and three trace to a sentence in the retrieved context, it scores 0.75 — whether or not the fourth claim happens to be true in the world. That distinction is the whole point: faithfulness asks "did the model stick to its sources," not "is the model right." A confident, plausible sentence with no support in the context is exactly what it's built to catch — because in RAG, that's the shape a hallucination takes.
Each model's faithfulness is averaged across the questions, with a 95% bootstrap confidence interval (2,000 resamples, fixed seed) so the small sample size is visible in the number rather than hidden behind it. When two models' intervals don't overlap, the difference is unlikely to be sampling noise; when they do, treat the gap as unproven.
Abstract metrics are easy to wave at, so here are two real questions from the eval set, run through the real pipeline.
A retrieval miss. Ask "What is HyDE, and when does it fail?" and the hybrid retriever's top 5 chunks come back as:
| # | Retrieved paper | Primary category |
|---|---|---|
| 1 | Evidence Attribution in Visual Document Understanding | cs.CV |
| 2 | Self-Study Reconsidered: Learning from Self-Generated Data | cs.AI |
| 3 | DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing | cs.AI |
| 4 | Self-Study Reconsidered (a second chunk) | cs.AI |
| 5 | DataOrchestra: Per-Example Curation of Pretraining Data | cs.CL |
Not one is about HyDE. The corpus is 100 recent papers and none of them defines it, so retrieval stays vaguely in the machine-learning neighborhood and misses. Now watch what the proxy does with that miss: the question's hint is cs.IR cs.CL, so category-precision counts the single cs.CL paper (#5) as a hit and the rest as misses — a score of 0.2. But #5 isn't about HyDE either, and the cs.AI papers it scored as misses might be more relevant or less — the metric can't tell. Whether it lands at 0.2 or 0.8, the number is blind to the one thing that matters: did retrieval surface a chunk that actually answers the question. "In the right topic" is all it can ever see.
A perfect faithfulness score for a non-answer. Now a question the corpus only mentions in passing — "How does Reciprocal Rank Fusion combine rankings?" Given the retrieved context, Claude Haiku answers:
"…the documents do not provide a detailed explanation of how RRF actually works mathematically or algorithmically… I don't know the detailed methodology based on the provided context."
The judge extracts five atomic claims from that reply (e.g., "RRF merges ranked lists from multiple retrieval channels," "the specific formula is not explained in the provided context") and finds all five supported by the retrieved chunks:
faithfulness = 5 / 5 = 1.00.
A perfect score — for an answer that never answered the question. And that's the metric working correctly: the retrieved chunks mentioned RRF but never gave its formula, so the model declined instead of inventing one, and every claim it did make was grounded. Faithfulness rewards not hallucinating; it says nothing about whether the question got a useful answer. Which is exactly why it's one axis of quality — the one that catches confident fabrication — and not the whole of it. (Getting the right chunk in front of the model in the first place is the retrieval problem above, and the reason Part 2 starts with gold labels.)
Here's the full mechanism behind the bug at the top of this post. My 33-question eval set carries topical source_hint categories but no gold chunk or document labels — nobody has marked "for this question, chunk X is the right answer." My first pass papered over that by deriving each question's "relevant" set from the retrieved results themselves — relevant ⊆ retrieved by construction — so recall was pinned at 1.0-or-0.0 and "precision" only measured category purity. Any recall or MRR computed that way grades the retriever against its own output. That leaves category-precision as the one signal I can defend, and even it only measures topical purity — not whether the right chunk was found.
| Config (k=5) | Category-precision@5 | 95% CI |
|---|---|---|
| A — dense | 0.23 | [0.14–0.33] |
| B — dense + rerank | 0.10 | [0.05–0.16] |
| C — hybrid (RRF) | 0.16 | [0.09–0.22] |
| D — multi-query | 0.21 | [0.13–0.29] |
| E — HyDE | 0.27 | [0.19–0.36] |
n = 27 scored of 33 questions · coarse topical-purity proxy — NOT relevance recall.
Read that as a diagnostic, not a leaderboard. Reranking (config B) scores lowest — but read that carefully rather than as "reranking is bad." A cross-encoder pulls the single most-relevant chunk to the top; if that chunk happens to sit in a neighbouring category, category-precision counts it as a miss. So a low score here can mean the reranker did its job, not that it failed — which is the whole problem: none of these numbers measure whether the right chunk was found, so none is a verdict on retrieval quality. Publishing 0.27 as a retrieval-quality score would be exactly the lie I caught myself in. Gold doc-level labels are the next eval task (Part 2); publishable retrieval claims wait for them.
This one is honest because the metric doesn't score itself. Hold retrieval constant — the same retrieved context per question — vary only the answer model, and score claim-level faithfulness with one fixed judge (GPT-4o-mini). Five models, three provider families, across two size tiers, on a 20-question sample — the first 20 of the 33, a fixed deterministic slice, chosen only to cap paid-model spend:
| Answer model | Provider | Faithfulness (n=20) | 95% CI |
|---|---|---|---|
| claude-haiku-4-5 | Anthropic | 0.83 | [0.77–0.90] |
| claude-sonnet-4-5 | Anthropic | 0.83 | [0.75–0.90] |
| gemini-2.5-flash | 0.33 | [0.16–0.51] | |
| gpt-4o-mini | OpenAI | 0.28 | [0.13–0.43] |
| gpt-4o | OpenAI | 0.20 | [0.08–0.34] |
n = 20 · judge: GPT-4o-mini (fixed) · faithfulness = supported claims ÷ total claims · 95% bootstrap CI. Latency is deliberately omitted — cached answers contaminate the timings, so a clean latency pass is a separate cold-cache run.
Three things this table says, and one it doesn't.
Claude is in a different grounding regime. Both Anthropic models cluster at 0.83, and their confidence intervals sit clear of every other model's — this is the one gap in the table that isn't sampling noise. Given identical retrieved context, Claude confines its answer to what that context supports far more consistently than the GPT or Gemini models do.
Bigger did not mean more faithful. Sonnet (0.83) is a dead heat with Haiku (0.83), and GPT-4o's point estimate (0.20) sits below GPT-4o-mini's (0.28) — though those two overlap within the sample, so read that as "no gain," not "GPT-4o is worse." Either way the direction is clear: whatever the flagship tiers buy — reasoning depth, breadth, polish — it isn't grounding on this task. If faithfulness is the axis you care about, the cheap tier is the buy; paying up bought no measurable grounding here.
The likely mechanism is the interesting part. Faithfulness measures grounding, not correctness. The GPT models tend to answer from what they already "know" about retrieval and RAG — often perfectly reasonable answers — but any claim the model supplies from its own training that the retrieved chunks don't contain scores as unsupported. Claude more often stays inside the provided context — the RRF example, where it flagged what the context didn't cover instead of filling the gap from memory, is that discipline in miniature. So read the table precisely: it does not say GPT-4o is a worse model. It says GPT-4o is a less grounded one — and for a RAG answerer meant to cite its sources, grounding is exactly the property you're paying for.
And it isn't Claude just hedging. The sharp objection to this whole table: a model that declines to answer trivially scores 100% faithful — the RRF non-answer above is exactly that — so maybe Claude wins by saying less. It doesn't. Counting the claims the judge extracts per answer, the Claude models make more of them, not fewer: about 6 (Haiku) and 8 (Sonnet) per answer, versus roughly 3 for GPT-4o-mini, GPT-4o, and Gemini. Claude is asserting twice as much and still grounding a higher share of it — saying more and staying honest, not saying less to protect the ratio.
And the one thing it doesn't say — the judge isn't playing favorites. GPT-4o-mini grades every answer, including its own family's, and it ranks OpenAI last. Whatever bias a single LLM judge carries, self-flattery isn't it here.
The caveats, stated plainly, because they're the point:
The corpus rebuilds from the committed manifest:
uv sync
uv run python scripts/run_ingest.py --max 100 # or rebuild the pinned IDs from data/manifest.jsonl
uv run python scripts/build_corpus.py # 100 papers → 11,187 chunks → ChromaDB (~115 MB)
uv run python tests/eval/run_ab.py # retrieval A/B (configs A–E)
uv run python tests/eval/run_llm_ab.py --n 20 # generation A/B (5 models)
One honesty note on the ingest line: ArXiv's "latest N" query drifts by the day, so --max 100 today fetches today's newest 100, not my pinned set. The reproducible artifact is data/manifest.jsonl — it pins all 100 ArXiv IDs, and the build downloads exactly those by ID. The query bootstraps the corpus; the manifest is the corpus.
Cost. The corpus build and the retrieval A/B run on free and local models — local BGE-small embeddings, local cross-encoder, ChromaDB, and the free Groq Llama default for the query-rewrite and HyDE steps → \(0. The generation A/B is the only paid run: five hosted models answering 20 questions each, plus a GPT-4o-mini judge extracting and verifying claims — an estimated ~\)0.30 (the repo doesn't meter tokens, so treat that as order-of-magnitude, not metered; GPT-4o and Claude Sonnet are the only non-trivial line items). Every LLM call is disk-cached, so re-running the identical eval is free. Running total stays inside the ~$15 project budget.
Part 2 turns this from a pipeline into an agent — tool-using, able to summarize a paper, discover citations, and find related work on its own. And before any retrieval-quality claim gets published, the eval gets gold doc-level labels, so the honest-but-limited category proxy above can finally become a real precision number.
The pattern I'm trying to hold to across all five parts: show the architecture, show the eval, show the cost — and when the eval can't back the claim, say so out loud instead of shipping the pretty table.
I'm a Technical Architect building AI-enabled systems. I write about what I build — the architecture and measurement decisions under the hood, not the model hype. The full project is open-source.
If you've caught your own eval measuring the wrong thing, I'd like to hear how. Find me on X @deepakgoyal_ai or LinkedIn.