Retrieval Augmented
Generation

How to give a language model access to a hundred thousand documents without ever putting a hundred thousand documents in the prompt.

Wil A decade in AI · incentivised data training with NEAR AI · retrieval and ingestion systems in production

Follow along at ragverse.diy — these slides, five playgrounds you can drive yourself, and all the code

RETRIEVAL LESSON · the theory, then the code
YOU NEED · Python 3.10+, an editor, one OpenAI API key
COST · Well under a cent of embeddings
EIGHT DEMOS · one per concept, each one runnable

Welcome. Before anything else, a quick word on who is talking. I am Wil. I have been working in AI for about a decade, which means I started well before GPTs made any of this fashionable — back when you built your own models and your own data pipelines because there was nothing to call. More recently I have been working on incentivised data training structures with NEAR AI: the mechanisms that reward people for contributing the data and the compute that models are actually trained on. And running through all of it, I have built a lot of retrieval and ingestion systems, on a lot of different stacks. I mention that for one reason. This is not a lesson I read and repeated. It is the shape of a system I have built more than once — and every mistake in it is a mistake I have made. So: we are going to build a Retrieval Augmented Generation system, and understand every piece of it. Here is the one-line version of what RAG is for. How do you give a language model access to a hundred thousand documents, without ever putting a hundred thousand documents into the prompt? That is the problem. Everything after it is mechanics. One practical thing before we start. Everything is at ragverse dot d i y — these slides, five interactive playgrounds you can drive yourself, and every line of the code. Open it in another tab now if you want to follow along. The address stays in the corner of every slide, so you do not need to write it down. Let's start with why the problem exists at all.

00 · Outcomes

What you will be able to do

Explain why a bigger context window does not remove the need for RAG.
Define token, context window, chunk, embedding, dimension, vector database, retriever and top k.
Draw both halves of a RAG system from memory.
Predict which chunks a retriever will return for a question, and say why.
Build a working ingestion pipeline in about sixty lines of Python.
Avoid the mistake that silently breaks most first RAG builds.

These are the outcomes. Six of them. By the end you will be able to explain why a bigger context window does not remove the need for RAG — that is the objection everyone raises first, and it has two answers. You will be able to define the vocabulary: token, context window, chunk, embedding, dimension, vector database, retriever, top k. These words get used loosely and that is where confusion comes from, so we will pin each one down as it appears. You will be able to draw both halves of a RAG system from memory. There are two halves and people routinely collapse them into one, which is where the mental model breaks. You will be able to look at a question and predict which chunks come back, and say why. You will have built a working ingestion pipeline — real code, on your machine, about sixty lines. And last, you will know the one mistake that silently breaks most first builds. Silently is the important word there. We will break it on purpose at the end so you know what it looks like.

Part one

Why RAG exists

The problem is not that models are stupid.
The problem is that they are small.

01 · The problem

Several hundred internal documents

Policy guidelines. Technical specs. Customer support logs. Contracts.

Someone asks a question that one of those documents answers. The obvious move is to paste all of them into the model and ask.

That does not work — and the reason it does not work is the whole reason this lesson exists.

01 · Definition

Retrieval Augmented Generation

A language model combined with a retrieval system. The retrieval system searches external sources — documents, databases, knowledge bases — and pulls only the relevant pieces into the prompt, at the moment the model needs them.

The model does not get everything. It gets the right few pages.

This is not a niche technique Almost every language-model product you have used is doing this underneath — a model that searches the web, an assistant that answers from your own files, a support bot that cites the article it used. Retrieval is the default way a model is given anything it was not trained on.

That is the entire idea. Everything else in this session is mechanics.

Here is the definition. Retrieval Augmented Generation is a language model combined with a retrieval system. The retrieval system searches external sources — documents, databases, knowledge bases — and it pulls only the relevant pieces into the prompt, at the moment the model needs them. Read that last part again, because it is the part people skip. At the moment the model needs them. Nothing is loaded in advance into the model. Nothing is trained into the model. The relevant text is fetched, per question, and placed in the prompt. So the model does not get everything. It gets the right few pages. That is the entire idea. I mean that literally — everything else in this session is mechanics for how you find the right few pages quickly. If you leave with only one sentence, leave with that one.

01 · Why anyone builds one

Where you have met one — and why

Where you have met it

The support assistant that answers and links the help article.
A model that searches the web and cites what it found.
A coding assistant that answers about your repository.
Internal Q&A over policy, security questionnaires, runbooks.

Why teams build it

Your data. No model was trained on your contracts or your tickets.
Today's data. A policy changed this morning. Index it; done.
Receipts. The retriever knows which chunks it used, so the answer can point at them.

The question everyone asks: why not fine-tune? Fine-tuning teaches a model a style — a format, a voice, a way of reasoning. It is an expensive way to teach it a fact, and a hopeless way to teach it a fact that changes on Tuesday. Retrieval gives a model facts it can point at.

Before any mechanics, the question underneath all of this: why does anyone build one? Start with where you have already met one, because you have. If you have used a support chatbot that answered your question and then linked you to the exact help article it got that from — that was retrieval. If you have watched a model search the web and cite what it found — retrieval. If you have asked a coding assistant something about your own repository and got a real answer rather than a plausible one — retrieval. Every company with an internal Q&A bot over its own policies is running one of these. Now the reasons teams build them. There are three, and they are worth separating. The first is your data. Whatever model you are using was not trained on your contracts, your runbooks, or your support tickets. It cannot know them. Ask it anyway and it will cheerfully invent something that sounds right. The second is today's data. A policy changed this morning. Somebody shipped a new product page an hour ago. You are not going to retrain a model because a policy changed — you drop the document in, and the next question picks it up. The third is receipts, and this is the one people underrate. The retriever knows exactly which chunks it used to answer, so the answer can point back at them. In most organisations an answer nobody can check is worth close to nothing. Being able to click through to the paragraph it came from is often the entire reason the project got approved. And then the question I get every single time: why not just fine-tune the model on our documents? Fine-tuning teaches a model a style. A format, a voice, a way of reasoning about a domain. It is a very expensive way to teach it a fact, and a hopeless way to teach it a fact that changes on Tuesday. It also gives you nothing to cite. Fine-tuning changes how a model speaks. Retrieval changes what it knows, and lets it show its working.

02 · Tokens

Models do not read words

A token is the unit of text a model processes. Sometimes a whole word. Often a fragment of one. Models do not read letters and they do not read words.

Counted with the cl100k_base tokenizer

text : 'Retrieval augmented generation is powerful' words : 5 tokens : 7 split as : Ret | rie | val | _augmented | _generation | _is | _powerful

One token is roughly three quarters of an English word. Common words are one token; rare or long words split into several. Spaces travel with the token that follows.

A word you will need: the token. A token is the unit of text a model processes. Sometimes it is a whole word. Often it is a fragment of one. Models do not read letters, and they do not read words — they read tokens. This matters because every limit and every price you will ever meet is denominated in tokens, not words. Here is what a real tokenizer does with a five-word sentence. Look at that. "Retrieval augmented generation is powerful" is five words. The model counts seven tokens. And look at how it split: "Retrieval" — a fairly ordinary English word — got broken into three pieces, R-e-t, r-i-e, v-a-l. Whereas "augmented", "generation" and "powerful" each survived as a single token. That is the rule in action. Common words are one token. Rarer or longer words get split into several. And those underscores are spaces — spaces travel with the token that follows them. The rule of thumb worth memorising: one token is roughly three quarters of an English word. So a thousand words is roughly thirteen hundred tokens.

02 · The context window

Everything has to fit inside one window

The context window is the total number of tokens a model can hold in a single request. Everything you send and everything it writes back has to fit. Past that limit, the information is simply not there.

Where the frontier sits As of 2026 the major models have converged on 1 million tokens — GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8 and Sonnet 5 all sit there. A few advertise up to 10 million, though no published benchmark shows quality holding near that.
Verify before you quote it. These figures move every few months.

The next word you will need: the context window. The context window is the total number of tokens a model can hold in a single request. And the critical detail is that everything counts toward it — everything you send in, and everything the model writes back out. They share one budget. Past that limit, the information is simply not there. Not summarised, not compressed. Not there. Right now, in 2026, the major models have converged on one million tokens. GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8, Claude Sonnet 5 — they all sit at a million. A few advertise ten million, though I'd treat that with some suspicion: no published benchmark shows answer quality holding anywhere near the top of those windows. Advertised capacity and usable capacity are not the same thing. Still — a million tokens is a genuinely enormous number. It is several very long books. I want to flag something about that figure: verify it before you quote it. These numbers move every few months, and a slide deck is exactly the kind of place a stale number goes to live forever. So — two million tokens. That sounds like it should be plenty. Let's see how it compares to how much text an actual company holds.

02 · The scale gap

Read the scale carefully

How much text, in tokens — logarithmic scale One chunk ███ 1,000 These 5 articles ████████ 72,169 A frontier model window ███████████ 1,000,000 The largest advertised ██████████████ 10,000,000 A mid sized company, 1 TB █████████████████████████ 250,000,000,000 An enterprise archive, 1 PB ██████████████████████████████████ 250,000,000,000,000

Each step to the right is ten times larger, not one step larger. The gap between a model window and an enterprise archive is not a gap you close by waiting for bigger models.

This is the same demo still running — the second half of demo one. Now, this is a logarithmic scale, and I need you to read it carefully, because logarithmic scales are quietly deceptive. Each step to the right is ten times larger than the last, not one step larger. The bars look comparable. The numbers are not. Start at the top. One chunk — that is a single retrievable piece of text, and we will define it properly in a moment — is about a thousand tokens. The five Wikipedia articles we are about to ingest come to seventy-two thousand tokens. That is our whole corpus for today, and notice — it comfortably fits inside a frontier model. For five documents you genuinely do not need RAG. I want to be honest about that rather than pretend otherwise. A frontier model window: one million. The largest advertised: ten million. Now watch what happens. A mid-sized company with one terabyte of documents: two hundred and fifty billion tokens. That is two hundred and fifty thousand times larger than the model window. An enterprise archive at one petabyte: two hundred and fifty trillion. So here is the thing to take away. That gap is not a gap you close by waiting for bigger models. If context windows got a thousand times bigger tomorrow — which they will not — you would still be short of the mid-sized company by a factor of two hundred and fifty. This is a structural problem, not a temporary one.

02 · Interactive

Try it: when do you actually need RAG?

Corpus scale explorer drag the slider

How much text do you have? 1.0 MB

tokens

chunks @800

to embed once

of a 1M window

Embedding cost assumes text-embedding-3-small at $0.02 per million tokens. One-off, not per question.

Let's make that scale argument something you can feel rather than just read, because the numbers are so large they stop meaning anything. Drag the slider. It runs from one megabyte of text up to one petabyte. [DRAG TO ~1 MB] At a megabyte we are at about two hundred and fifty thousand tokens. That fits in a frontier window with room to spare, and the panel says so — you do not need RAG for this. Be honest about that. If your entire corpus is a handful of documents, retrieval is overhead, not architecture. [DRAG TO ~1 GB] One gigabyte. Two hundred and fifty million tokens. Now we need over a hundred context windows. There is no prompt you can write that holds this. [DRAG TO 1 TB] And a terabyte — a mid-sized company — needs a hundred and twenty-five thousand full context windows. Watch the cost figure while you drag, because that is the one people do not think about. Embedding a terabyte once costs about five thousand dollars. That is a one-off, not per question — but it is a real number, and it is why you choose your embedding model carefully before you start rather than after. The threshold where RAG stops being optional is lower than people expect. Somewhere around a few megabytes, this stops being a choice.

02 · The second reason

You also pay per token

Even when everything does fit, sending it all is the wrong move.

Send everything

Expensive — you pay for every token
Slow — latency scales with input
Worse answers — the signal is buried

Send the right five paragraphs

Cheap
Fast
Better answers

The point Sending 500,000 tokens of irrelevant context to answer one question is expensive, slow, and produces worse answers than sending the five right paragraphs.

There is a second reason, and it matters just as much as the first, but people forget it because the first one is so dramatic. Even when everything does fit — even in the case where your documents are small enough — sending all of it is still the wrong move. Three reasons. One, you pay per token, so you are paying for every irrelevant word. Two, it is slow; latency scales with how much you send. And three — this is the one that surprises people — you get worse answers. That third point is counterintuitive so let me be explicit about it. If you bury the one relevant paragraph inside five hundred thousand tokens of unrelated material, the model has a harder job finding it than if you had simply handed it the paragraph. More context is not more helpful. Relevant context is helpful. So: sending five hundred thousand tokens of irrelevant context to answer one question is expensive, slow, and worse. Sending the five right paragraphs is cheap, fast, and better. That is the case for RAG, complete. Now let's look at how it is actually built.

Part two

The shape of the system

Two pipelines, not one.
Learn them separately and the rest is easy.

03 · The whole system in one picture

Two pipelines

Pipeline 01 · INGESTION · runs once, before any question

Sourcedocuments

→

Chunkingcut to size

→

Embeddingmodel

→

Vectorslists of numbers

→

Vectordatabase

Pipeline 02 · RETRIEVAL · runs on every question

Userquestion

→

Embeddingmodel

→

Retrieverranks by closeness

→

Top matchingchunks

→

LLM writesthe answer

Not a drawing shortcut The same embedding model appears in both rows. That is the single most important detail on this slide, and we will break it on purpose at the end to show you why.

Here it is. The whole system in one picture. The top row is the ingestion pipeline, and the thing to understand about it is that it runs once, ahead of time, before anybody asks anything. Source documents go in. They get chunked — cut into small pieces. Each piece goes through an embedding model, which turns it into a vector, a list of numbers. Those vectors get stored in a vector database. Done. That pipeline does not run again until your documents change. The bottom row is the retrieval pipeline, and it runs every single time someone asks a question. The question comes in. It goes through an embedding model and becomes a vector. A retriever compares that vector against everything in the database and ranks by closeness. The top matching chunks come back. And those chunks, plus the question, go to the LLM, which writes the answer. Now look at the two orange boxes. The embedding model appears in both rows. That is not me being lazy with the drawing. It is the same model, and it has to be the same model. If those two boxes ever contain different models, the entire system fails — and it fails silently, with no error message. We are going to break that rule deliberately in the last demo so you know exactly what it looks like. Keep this diagram. Every RAG system you will ever build, from a weekend project to production, is this diagram with more engineering around each box.

04 · Chunking

A chunk is a slice, not a section

Chunking is breaking large documents into small pieces. You choose the size. Set it to 1,000 tokens and 10 million tokens of documents becomes 10,000 chunks.

The word "chunk" is doing work here Chunks are cut by size, not by meaning. They do not respect chapters, headings or arguments. A naive splitter will cut through the middle of a sentence or a table without hesitating.

Why cut at all? Because nobody wants an entire Wikipedia article returned because one sentence in it matched. The chunk is your unit of retrieval — it is the smallest thing the system can hand back.

Next word: the chunk. Chunking is breaking large documents into small pieces. You choose the size. If you set it to a thousand tokens, then ten million tokens of documents becomes ten thousand chunks. Simple arithmetic. Now, the word "chunk" is doing some work here and I want to be precise. A chunk is a slice, not a section. Chunks are cut by size, not by meaning. They do not respect chapter boundaries, headings, or the structure of an argument. A naive splitter will cut straight through the middle of a sentence, or a table, without hesitating. You will see exactly that when we get to the chunking step. So why cut at all? Why not keep whole documents? Because the chunk is your unit of retrieval — it is the smallest thing the system is able to hand back. If your chunks are whole Wikipedia articles, then a question about one sentence returns the entire article, and you are back to stuffing the context window with mostly-irrelevant text. Which is the problem we started with. Chunk size is therefore a real design decision, and there are techniques for cutting more intelligently. Those come later in the track. For today we cut simply, and I will show you exactly what that costs.

04 · The embedding model

This is not an LLM

Different model, different job. It does not generate text. Text goes in; a list of numbers comes out.

Measured with text-embedding-3-small

one word 1 words in -> 1536 numbers out one sentence 8 words in -> 1536 numbers out a paragraph 337 words in -> 1536 numbers out [ -0.0616 +0.0554 +0.0004 +0.0124 +0.0016 -0.0123 ... ]

Fixed length output. One word in or nine hundred words in, the vector that comes out is always the same length.

Why it is always the same length

The model reads the text once and produces one vector per token, then pools them into a single vector. Pooling is what throws the length away.

Three properties worth knowing

Unit length. Every vector comes back normalised, so cosine similarity is just a dot product.
8,192 token cap. Longer input is truncated silently — another reason to chunk.
The dimensions are ordered. Keep the first 512 and renormalise and it still works — a third of the storage.

Next box in the diagram: the embedding model. And the first thing to say, because it trips everybody up, is that this is not an LLM. Different model, different job. It does not generate text. It does not chat. You give it text, and it gives you back a list of numbers. That is all it does. Look at the three rows on screen. One word in. An eight-word sentence in. A three-hundred-and-thirty-seven-word paragraph in. And every single time, what came back was fifteen hundred and thirty-six numbers. Not more for the paragraph. Not fewer for the single word. Always exactly fifteen thirty-six. Underneath you can see the first few actual numbers. They are small, they are signed, and — this is worth saying plainly — no individual number there means anything you can name. Nobody knows what dimension four hundred and twelve represents. That is fine. It is not how they are used. The property that matters is the fixed length. One word in or nine hundred words in, the vector is the same size. That is the property the entire system is built on: any two pieces of text, however different in length, become two lists of the same size — and two lists of the same size can be compared with simple arithmetic. That is the trick. Everything downstream depends on it.

05 · What an embedding is

Similar meaning, similar numbers

A vector embedding is a list of numbers that stands in for a piece of text. Each number is called a dimension.

Text with similar meaning produces similar numbers.
Nothing else about embeddings matters as much as that sentence.

Word	Dim 1 · size	Dim 2 · domesticated	Dim 3 · sound
cat	34	8	7.5
kitten	33	8	7.1
dog	40	8	6.9
elephant	210	62	1.2

Real dimensions are not labelled and nobody knows what any single one means. These labels are a teaching device only.

So what is an embedding, actually? This is the part people get stuck on, so we will slow down here. A vector embedding is a list of numbers that stands in for a piece of text. Each number in that list is called a dimension. Two more words for the list — and dimension just means one position in it. And here is the sentence that matters more than anything else in this session: text with similar meaning produces similar numbers. It is worth being concrete about why that is the whole ballgame. Somebody types "my card got declined". The help article that answers them says "payment failures". Not one word in common. A keyword search — the thing every support site had for twenty years — returns nothing at all, and the customer gives up. An embedding model puts those two pieces of text next to each other, because it was trained on how words are used rather than how they are spelled. Matching on meaning instead of spelling is the entire thing you are buying here. That is the property. That is what an embedding model is trained to do. Now, this table is a teaching device, and it is a fake. I have made up three dimensions and labelled them size, domesticated, and sound. Real embeddings do not work like this — the dimensions are not labelled, and nobody knows what any individual one means. But the fake is useful for building intuition, so read it column by column. Cat and kitten sit almost on top of each other on every dimension — thirty-four and thirty-three on size, both eight on domestication, seven point five and seven point one on sound. Because a kitten is a young cat. Dog shares the domestication value but sits further out on size. And elephant is far away from all three on everything — two hundred and ten on size against thirty-four. That is the intuition. Now let's check whether it survives contact with a real embedding model, because I do not want you taking a made-up table on faith.

05 · Eight real words

Does that actually hold?

Real embeddings · cosine similarity · text-embedding-3-small

cat -> dog 0.603 ████████████████████████ cat -> kitten 0.570 ███████████████████████ cat -> coffee 0.385 ███████████████ cat -> tea 0.358 ██████████████ cat -> apple 0.357 ██████████████ cat -> elephant 0.319 █████████████ cat -> mango 0.264 ███████████ DOMESTIC ANIMALS inside 0.531 outside 0.316 DRINKS inside 0.611 outside 0.313

Every group is tighter inside than out. That is a neighbourhood — and nobody labelled any of it.

So does that claim actually hold? I did not want to assert it and move on, so here it is measured. This takes eight ordinary words, embeds each one with a real OpenAI model, and measures how close every pair is. One point zero would be identical. Top of the list: cat to dog, zero point six oh three. Then cat to kitten at zero point five seven. Then a clear drop — coffee, tea, apple, elephant, mango, all down in the threes and twos. Now, I want to point at something honest here, because it contradicts the tidy table on the previous slide. That table said kitten should be closest to cat. The real model puts dog first, with kitten second. So the toy intuition was directionally right and specifically wrong. Why? Because bare single words are a weak signal. "Cat" and "dog" co-occur in text constantly — they are the two canonical pets — whereas "kitten" is a narrower word. Real systems almost never embed single words; they embed whole paragraphs, where there is far more meaning to work with. So do not over-read the ordering inside a cluster. But look at the bottom two lines, because that is where the claim really holds. Domestic animals: average similarity inside that group is zero point five three, and to everything outside it, zero point three two. Drinks: zero point six one inside, zero point three one outside. Every group is meaningfully tighter inside than out. That is a neighbourhood. And the crucial part — nobody labelled any of this. No human told the model that coffee and tea are related. It fell out of arithmetic over fifteen hundred numbers.

05 · Interactive

Try it: the neighbourhood map

Real embeddings · text-embedding-3-small click any word to make it the probe

cat

probe word

—

closest

—

furthest

These are genuine 1,536-dimension OpenAI vectors, precomputed by tools/export_slide_data.py. The map is a 2-D projection of all 1,536.

Now you can play with it yourself. These are real vectors — genuine fifteen-hundred-dimension OpenAI embeddings, computed ahead of time and shipped with the slide so it works without an API key. Click any word and it becomes the probe. Everything re-ranks against it. On the left, similarity as bars, coloured by category — orange for animals, green for fruit, purple for drinks. On the right, a two-dimensional projection of all fifteen hundred and thirty-six dimensions. That projection throws away most of the information, so treat it as a sketch rather than the truth — but the clustering it shows is real. [CLICK 'coffee'] Look — tea comes straight to the top. Water is up there too. The animals fall away. [CLICK 'puppy'] Puppy. Dog first, obviously. Then kitten, then cat. It has learned that puppy-dog and kitten-cat are the same relationship, without anybody telling it what a young animal is. [CLICK 'lion'] And here is a nice one. Lion sits near cat and elephant but away from dog — it has picked up something about wild versus domestic that we never labelled. Try a few. The thing I want you to notice is that every ordering you see came out of arithmetic on numbers, and every one of them is defensible. That is what "similar meaning gives similar numbers" actually looks like in practice.

06 · Choosing a model

The ones you will meet first

Model	Default dimensions	Notes
`text-embedding-3-small`	1,536	Fine for most projects. Cheap. We use this one.
`text-embedding-3-large`	3,072	Best quality of the two. Costs more.
`voyage-3-large`	1,024	Strong on technical and code text.
`Cohere embed`	1,024	Good multilingual support.
`Mistral embed`	1,024	Open weights available.

A cheap win Most of these let you request fewer dimensions than the default. Asking a 3,072 model for 512 cuts your storage by six and usually costs very little accuracy.
Check current pricing and dimension options in the provider docs. These move.

There are many embedding models. These are the ones you will meet first. Text-embedding-3-small from OpenAI, fifteen hundred and thirty-six dimensions. Cheap, and fine for most projects. That is the one we are using today, and I want you to notice that I am telling you the model and the dimension count in the same breath. That habit will make sense shortly. Text-embedding-3-large, three thousand and seventy-two dimensions. Better quality, costs more. Then Voyage, which is strong on technical and code text — worth knowing if you are indexing a codebase. Cohere, good multilingual support. And Mistral, which has open weights if you need to run it yourself. One genuinely useful thing at the bottom. Most of these models let you request fewer dimensions than the default. If you ask a three-thousand-dimension model for five hundred and twelve dimensions, you cut your storage by a factor of six and typically lose very little accuracy. That is a real lever in production and most people do not know it exists. Same caveat as the context windows: check current pricing and dimension options in the provider documentation before you commit. These move. There is a trade-off underneath all of this. More dimensions capture more meaning, and also cost more to compute and more to store. Most production systems reduce dimensions deliberately.

07 · Where the vectors live

The vector database stores both

ID	Vector	Original text	Source
0001	`[0.021, -0.884, ...]`	Refunds are issued within 30 days.	`policy.pdf`
0002	`[-0.412, 0.663, ...]`	Q1 revenue reached 4.2 million.	`sales.xlsx`
0003	`[0.907, 0.115, ...]`	Guest wifi resets every Monday.	`itwiki.md`

One row per chunk The vector is how it gets found. The original text is what gets used. Store only the numbers and you have nothing to send the model.

Purpose built: Pinecone, Weaviate, Chroma, Qdrant. FAISS is a library rather than a service. Or add pgvector to a Postgres you already run.

Last box in the ingestion row: the vector database. A vector database stores embeddings and finds the closest ones to a given vector, fast. That is its specialty — not storing things, but searching them by proximity across a huge number of dimensions. But here is the detail people miss, and it is the reason I put this table up. Crucially, it stores the original text alongside each vector. Look at the columns: an ID, the vector, the original text, and the source file. One row per chunk. The vector is how it gets found. The original text is what gets used. If you store only the numbers, you have a beautiful search index and absolutely nothing to send to a language model — because you cannot turn a vector back into English. It is a one-way trip. The fourth column, source, is metadata. That is what lets you tell a user "this answer came from policy dot pdf, page four". We will watch that column survive all the way through the pipeline. For options: Pinecone, Weaviate, Chroma, and Qdrant are purpose-built vector databases. We are using Chroma today because it runs locally, on disk, with no account and no server. FAISS is a library from Meta rather than a hosted service. And if you already run Postgres, the pgvector extension turns it into a vector database, which is often the right answer in a real company.

Part three

Retrieval, in principle

Ingestion is finished. Nothing above runs again
until your documents change.

08 · Retrieval, step by step

The question takes the same road

The question goes through the same embedding model the documents went through.
It comes out as a vector of the same length.
The retriever compares that one vector against every stored vector.
It ranks them by closeness and returns the top k.

— worked example, not a program run — THE QUESTION "What were our sales in the first quarter?" ↓ [-0.398, 0.671, 1.194, ...] same model, same dimensions 0.91 Q1 revenue reached 4.2 million. ┐ 0.88 Quarterly sales by region, Jan to Mar. ├ top 3 to the LLM 0.84 Revenue targets for the first quarter. ┘ 0.42 Annual headcount summary. 0.31 Refunds are issued within 30 days. 0.19 Guest wifi resets every Monday.

Here is the whole retrieval pipeline in four steps. Step one. The question goes through the same embedding model the documents went through. Same model. I am going to keep repeating that until it is annoying, because it is the thing that breaks. Step two. It comes out as a vector of the same length — fifteen hundred and thirty-six numbers, exactly like every chunk in the database. Step three. The retriever compares that one vector against every stored vector. Conceptually every one; in practice the database uses an index to avoid a brute-force scan, but the effect is the same. Step four. It ranks them by closeness and returns the top k. Look at the example. This one is drawn, not run — it says so at the top — because I want a tidy invented company here rather than five Wikipedia articles. The point is easier to see. The question is "what were our sales in the first quarter". It becomes a vector. And then every chunk gets a score. Q1 revenue reached four point two million: zero point nine one. Quarterly sales by region: zero point eight eight. Revenue targets for the first quarter: zero point eight four. Those three go to the model. And then look at what did not make it. Annual headcount summary at zero point four two — related to business, not to the question. Refunds at zero point three one. Guest wifi at zero point one nine, which is about as unrelated as it gets. Notice something about that top result. The question says "sales". The winning chunk says "revenue". Those are different words. A keyword search for "sales" would have missed it entirely. That is what embeddings buy you — matching on meaning rather than on spelling. How closeness is actually calculated is a later topic. For now: it is distance between two points, measured with arithmetic.

08 · Top k

You choose how many come back

That number is called top k. Ask for the top 5 and you get 5 results — whether or not all 5 are any good.

A retriever always returns something It has no concept of "no good match". If your corpus contains nothing relevant, you still get k results back, ranked. They are just the least-bad of a bad set. Handling weak matches is a later topic — but knowing it happens is not.

So: a retriever that returns 10 chunks of which 4 are irrelevant is not broken. That is the expected shape of the output.

The number of chunks that come back is called top k. That is the last of the words worth collecting. You choose it. Ask for the top five and you get five results. And now the important half of that sentence: whether or not all five are any good. A retriever always returns something. It has no concept of "there is no good match here". If your entire corpus contains nothing relevant to the question, you still get k results back, neatly ranked, looking perfectly respectable. They are simply the least-bad of a bad set. This catches people out constantly. Somebody builds their first RAG system, asks it a question the documents genuinely do not answer, gets five confident-looking chunks back, and concludes the retriever is broken. It is not broken. That is the expected shape of the output. So if a retriever returns ten chunks and four are irrelevant — that is normal. Filtering on a score threshold, or re-ranking, is how you handle it, and that is a later topic in the track. But knowing that it happens is not a later topic. That is today.

08 · The step everyone misreads

Vectors find. Text is what gets sent.

Sent to the model

The user's question
The original English text of the top chunks

Not sent

The vectors
Anything numeric at all

Vectors are only used for finding. After retrieval you are back in plain English. The prompt that reaches the model is the question and a few paragraphs of ordinary text.

This is the step everyone misreads, so I am giving it its own slide. There is a very common mental model in which the vectors somehow go to the language model — as if the LLM understands embeddings, or as if the numbers are a compressed form the model can read. That is wrong, and it is worth killing off explicitly. What is sent to the model: the user's question, and the original English text of the top chunks. That is it. What is not sent: the vectors. Anything numeric at all. Vectors are only used for finding. Their job is over the instant the matching is done. After retrieval you are back in plain English, and the prompt that reaches the model is the question plus a few paragraphs of perfectly ordinary text — the kind of thing you could have pasted in by hand. Once that clicks, a lot of RAG stops being mysterious. The clever part is the search. The prompt at the end is boring, and it is meant to be. We are going to print that exact prompt to the screen later so you can read every character of it. Right. That is all the theory. Let's build it.

Part four

What actually happens to your documents

Five articles in. A searchable vector database out.
About sixty lines of Python and one API key.

Five articles go in. A searchable database comes out.
Three steps — and what each one actually produces.

09 · Where this sits

Orienting you before we start typing. This is the same two-row diagram, with the concrete numbers for today filled in. Five text files in a docs folder. They become five hundred and forty-seven chunks. Those go through OpenAI's embedding model and become five hundred and forty-seven vectors, which land in a Chroma database on disk. The bottom row — question, embed, retrieve, build a prompt, answer — is the next lesson. We will touch it briefly at the end today just to prove the database works, but building it properly is next time. And again: ingestion runs once. When we finish, you will not run this code again unless your documents change. That is worth internalising because it shapes how you think about cost. The expensive step happens once, up front, and every question after that is cheap.

10 · The documents

Five articles

google.txt 68,012 characters 180 paragraphs microsoft.txt 67,411 characters 173 paragraphs nvidia.txt 62,402 characters 246 paragraphs spacex.txt 55,954 characters 172 paragraphs tesla.txt 93,916 characters 339 paragraphs TOTAL 347,695 characters

Any five text files will work. Company articles are convenient because they are long, factual, and full of specific numbers you can test the retriever against.

You need documents. For this walkthrough I used the Wikipedia articles for Google, Microsoft, Nvidia, SpaceX and Tesla, saved as plain text — about three hundred and fifty thousand characters across five files. Any five text files work. Company articles are convenient for teaching because they are long, factual, and full of specific names and numbers you can test the retriever against. When you ask "who founded SpaceX" you know what the right answer is, so you can tell immediately whether retrieval worked. One detail worth mentioning, because it is the kind of thing that quietly ruins a pipeline. Wikipedia's plain-text export separates paragraphs with a single newline. The splitter we are about to use splits on blank lines. So the fetch script normalises every paragraph onto its own blank-line-separated block. Skip that step and the splitter finds almost no break points and produces eight-thousand-character chunks, which are useless units of retrieval. That is a real data-cleaning step, and it is invisible until you look at your chunk sizes.

11 · Step one: reading the documents

A document is its text, plus where it came from

Loaded 5 documents google.txt 68,012 characters microsoft.txt 67,411 characters nvidia.txt 62,402 characters spacex.txt 55,954 characters tesla.txt 93,916 characters .page_contentthe text 'Google LLC ( , GOO-gəl) is an American multinational...' .metadataits source {'source': 'docs/google.txt'}docs/google.txt

Learn these two attributes now .page_content is the entire text of the file as one long string. .metadata is a dictionary — here, the file we read it from. This object shows up everywhere from here on. Two halves, and you need both Every record is the text itself — the whole file as one long run of characters — and where it came from. The second half is what lets an answer cite its source.

Here is what comes back. Five files in, five Document objects out. That Document object shows up everywhere from here on, so learn its two attributes now — there are only two and they are both simple. Dot page underscore content is the entire text of the file as one long string. Not a list of lines, not a stream. One string — sixty-eight thousand characters for Google, ninety-four thousand for Tesla. Dot metadata is a dictionary. Right now it holds one key, source, pointing at the file path — we put it there ourselves, which is worth noticing, because it means you can put anything you like in it. You can add your own keys later — page numbers, authors, dates, permissions — and that is how real systems do access control and citation. And here is the property that makes metadata worth caring about: metadata survives everything. When a document gets chunked in a moment, every single chunk inherits its parent's metadata. That is how a RAG system can tell a user which file an answer came from, even though the thing it retrieved was a four-hundred-character fragment.

12 · Two gotchas

Two things worth knowing

Why sorted() is not decoration DirectoryLoader walked the directory in whatever order the filesystem returned — so index 0 changed between machines, and chunk order changed with it. Sorting makes the run reproducible. Either way, never write code that assumes index 0 is a particular file; filter on the metadata.

Text is the easy case PDFs, spreadsheets and HTML need a real parser, and that is worth a dependency: langchain-unstructured, or pypdf directly. What comes out is the same thing — Document objects — so everything downstream is unchanged.

Two things worth knowing. First, that call to sorted is not decoration. DirectoryLoader walked the directory in whatever order the filesystem handed back — which meant index zero could be a different file on your machine than on mine, and the chunk order changed with it. For a lesson where we compare numbers, that is genuinely annoying. Sorting makes the run reproducible. But the underlying advice survives either way: never write code that assumes index zero is a particular file. If you need a specific document, filter on the metadata. Second, text is the easy case. I have just told you that you do not need a loader package, and for dot-txt files that is true. PDFs, spreadsheets and HTML are a different story — those need a real parser, and that is absolutely worth a dependency. Reach for langchain-unstructured, or pypdf directly. The important part is that what comes out the other side is the same thing: Document objects, with page content and metadata. So everything downstream of this box — chunking, embedding, storing, retrieving — does not change at all.

13 · Step two: cutting them up

800 is a target, not a cap

Split 5 documents into 539 chunks chunks 539 smallest 15 characters largest 1,719 characters average 643 characters 79 of 539 chunks came out longer than the 800 target

Why CharacterTextSplitter splits on the separator first, then merges pieces up to chunk_size. It never breaks a paragraph in half to hit the number. So you will see warnings — they are warnings, not errors.
RecursiveCharacterTextSplitter handles this better and is what you would use in a real project. We stay on the simple one so the mechanics show. The splitter breaks on blank lines first, then glues the pieces back together up towards your target. What it will never do is cut a paragraph in half to hit the number — so a long paragraph gives you a long chunk.

Five hundred and thirty-nine chunks, from five documents. Now look at the spread, because this is where the mental model needs correcting. The smallest chunk is fifteen characters. The largest is one thousand seven hundred and nineteen. The average is six hundred and forty-three. We asked for eight hundred and got almost nothing that is actually eight hundred. Seventy-nine of the five hundred and thirty-nine came out longer than the target. Here is why. CharacterTextSplitter splits on the separator first — blank lines, in our case — and then merges the resulting pieces back together up to chunk size. What it will not do is break a paragraph in half to hit the number. So if a single paragraph is seventeen hundred characters, you get a seventeen-hundred-character chunk, and LangChain logs a warning for each one. Those warnings are warnings, not errors. I have suppressed them in the demo and counted them instead, because otherwise seventy-nine lines scroll past and you cannot see anything else. But you will see them if you write this yourself, and now you know they are expected. Chunk size is a target, not a hard cap. RecursiveCharacterTextSplitter handles this better — it tries a sequence of separators, falling back to smaller ones — and it is what you would reach for in a real project. We are staying on the simple one today so that the mechanics stay visible.

13 · Overlap

What overlap actually does

chunk_overlap = 0overlap = 0 characters [0] Tesla was incorporated in July 2003 by Martin Eberhard and Marc Tarpenning as Tesla Motors. Its name is a tribute to the [1] inventor Nikola Tesla. In February 2004 Elon Musk led Tesla's... chunk_overlap = 40overlap = 40 characters [0] Tesla was incorporated in July 2003 by Martin Eberhard and Marc Tarpenning as Tesla Motors. Its name is a tribute to the [1] Motors. Its name is a tribute to the inventor Nikola Tesla. In...

With overlap at zero, a sentence that straddles the boundary ends up half in one chunk and half in another — and neither half matches the question well.

Rule of thumb: 10–20% of chunk size. For 800 characters, 100 is a reasonable default.

This is the part of demo five I actually care about, and it is shown on a short passage with a small chunk size so the whole thing fits on one screen. The mechanism is identical at eight hundred characters. Top block, overlap zero. Read the seam between chunk zero and chunk one. Chunk zero ends with "its name is a tribute to the". Chunk one begins with "inventor Nikola Tesla". The sentence is cut clean in half. Think about what that costs you. Neither half carries the whole fact. If someone asks "who is Tesla named after", chunk zero has the question's subject but not the answer, and chunk one has the answer but has lost the subject. Neither one is a good match. The fact is in your database and it is unreachable. Bottom block, overlap forty. Now chunk one begins by repeating the tail of chunk zero — "Motors, its name is a tribute to the" — and then continues into "inventor Nikola Tesla". The sentence on the seam survives intact inside one chunk. It is retrievable again. That is all overlap does. Each chunk repeats the last n characters of the one before it. Rule of thumb: set overlap to roughly ten to twenty percent of chunk size. For eight hundred characters, one hundred is a reasonable default. You pay for that duplication in storage and in embedding cost — the same text gets embedded twice — and it is almost always worth it. So we change the default to a hundred, and re-run.

14 · Interactive

Try it: watch the chunks re-cut

Chunking playground overlap is highlighted in each chunk

chunk_sizechunk size 400 chars

chunk_overlapoverlap 0 chars

chunks

avg chars

duplicated

est. tokens

This is the same two numbers you just saw in code, except now you can move them and watch what they cost you. The document is the opening of the Tesla article — the same text demo five used. [DRAG chunk_size DOWN TO ~150] Small chunks. Look at the count climbing and the average dropping. These are precise — each one is about a single thing — but read them. They have lost their surroundings. A chunk that says "in 2008 he was named chief executive officer" does not say who "he" is. Retrieve that on its own and it is useless. [DRAG chunk_size UP TO ~1200] Now large chunks. Only a couple of them. Each one carries plenty of context — but now a question about the Roadster matches a chunk that is mostly about something else, and you are sending three times as many tokens to answer it. The match gets diluted. [DRAG chunk_overlap UP TO ~100] Now watch the highlight. That green text at the start of each chunk is repeated from the one before it. Look at the "duplicated" figure climbing — that is text you are storing twice and paying to embed twice. [SET overlap TO 0, THEN BACK TO 100] And that is the trade. At zero you pay nothing extra and sentences on the seam get cut in half. At a hundred you pay maybe fifteen percent more and they survive. There is no correct answer on this slide. There is a shape of answer: chunks big enough to stand alone, small enough to be about one thing, with enough overlap that the seams do not eat your sentences.

15 · Step three: turning chunks into vectors

Every chunk becomes numbers, text and a source

Stored 547 vectors in db_chroma/ the collection now holds 547 vectors the vector 1,536 numbers [+0.0223, -0.0135, -0.0015, ...] the original text 'Google LLC ( , GOO-gəl) is an American multinational...' the metadata {'source': 'docs/google.txt'}

db_chroma/
├── chroma.sqlite3          text, metadata, ids
└── a1b2c3d4-.../           the collection
    ├── data_level0.bin     the vectors
    └── link_lists.bin      the search index

The original text is not optional. Without it you have numbers and nothing to send an LLM.

Five hundred and forty-seven vectors stored. And a folder has appeared that was not there a moment ago. Look at what is stored per chunk, because this is the vector database table from earlier, now real. The vector: fifteen hundred and thirty-six numbers. The original text: the opening of the Google article. And the metadata: source, docs slash google dot txt. There is the metadata surviving, exactly as promised. It went in at the loader, came through the splitter untouched, and is now sitting in the database attached to a chunk. Underneath, the folder structure. Chroma dot sqlite three holds the text, the metadata and the ids — it is an ordinary SQLite database, you can open it. And then a folder named after the collection's UUID holds the binary files: data level zero dot bin is the vectors themselves, and link lists dot bin is the search index, which is what makes proximity search fast rather than a brute-force scan of half a million numbers. And the line to remember: the original text is not optional. Without it you have numbers and nothing to send an LLM. That is why this database stores both.

16 · Retrieval, for real

Ask it who founded SpaceX

Searched across 547 chunks of five real articles

question 'Who founded SpaceX?' collection 547 vectors 0.662 ███████████████ spacex.txt In early 2002, Elon Musk started to look... 0.628 ██████████████ spacex.txt SpaceX was founded in 2002 by Elon Musk... 0.627 ██████████████ spacex.txt Space Exploration Technologies Corp., ... 0.598 █████████████ spacex.txt The company is credited with advances... 0.591 █████████████ spacex.txt In early 2012, approximately two-thirds...

All five from spacex.txt, out of a store containing four other companies. Your ingestion pipeline is finished and correct.

So does it work? Here is the store being asked a real question. The question is "who founded SpaceX". The collection has five hundred and forty-seven vectors in it, covering five different companies. And every single result comes from spacex dot txt. Not one Tesla chunk, not one Microsoft chunk — despite Elon Musk appearing prominently in the Tesla article, which is exactly the kind of near-miss that would trip a keyword search. Look at the top hit: "In early 2002, Elon Musk started to look for staff for his company, soon to be named SpaceX." That answers the question. Score zero point six six two. One thing to note about the scores. Zero point six six is the top match, and that might feel low if you were expecting something near one. It is not low. Cosine similarity between a short question and a long paragraph rarely goes above zero point seven, because they are different shapes of text. What matters is the gap between the top results and everything else, not the absolute number. Do not go hunting for a universal threshold — calibrate against your own corpus.

17 · The actual prompt

This is what reaches the model

│ Answer the question using only the context below. │ │ CONTEXT: │ In early 2002, Elon Musk started to look for staff for his company, │ soon to be named SpaceX. Musk approached five people for the initial │ positions, including Griffin, who declined the position of Chief │ Engineer, Jim Cantrell and John Garvey ... │ │ SpaceX was founded in 2002 by Elon Musk with the goal of reducing │ spaceflight costs ... │ │ QUESTION: Who founded SpaceX? prompt length: 2,972 characters, built from the top 3 chunks

Sent: the question, plus the original English text. Not sent: the vectors. Their job finished the moment the matching was done.

And here it is. The actual prompt. Every character of it, printed by the demo. I promised earlier we would look at this, because it is the step everyone misreads. Read what is on that screen. An instruction: "answer the question using only the context below." Then the word CONTEXT, and then three paragraphs of completely ordinary English lifted straight out of the Wikipedia article. Then the word QUESTION, and the user's question. That is it. That is the whole thing. Just under three thousand characters. There is nothing numeric anywhere in it. No vectors, no embeddings, no similarity scores. The model has no idea a vector database was involved. As far as it is concerned, somebody pasted three paragraphs and asked a question. Vectors are only used for finding. Their job finished the moment the matching was done, and then we went back to plain English. I think this is the single most clarifying thing in the whole lesson. All the machinery — the chunking, the embedding, the fifteen hundred dimensions, the HNSW index — exists to produce those three paragraphs. The clever part is the search. The prompt at the end is boring, and it is supposed to be.

17 · Interactive

Try it: ask, and watch the prompt build

Retrieval playground pick a question

question

top k 2 chunks

prompt tokens

document tokens

sent to model

0.00

top score

The four preset questions were embedded ahead of time with text-embedding-3-small, so those scores are real. A question you type cannot be embedded in a browser — that needs an API call — so it falls back to a rough local measure, and the panel says which one you are looking at.

Last interactive one, and it puts the whole retrieval pipeline in your hands. Small policy document, cut into chunks. Type a question, and every chunk gets scored against it. The highlighted ones are the ones that would be sent. One honesty note before I start clicking. The four preset questions were embedded ahead of time with the real model, so those scores are genuine cosine similarity over fifteen hundred dimensions. If you type your own question, the page cannot embed it — that needs an API call — so it falls back to a rough local measure. The panel tells you which one you are looking at, and you should watch that label. [CLICK 'laptop budget'] Watch the ranking jump. The equipment chunk goes to the top and the prompt underneath rebuilds itself in real time. [DRAG top k FROM 2 TO 6] Now raise top k. More chunks get highlighted, and watch the "sent to model" figure climb — by six it reads a hundred percent. Every part of the document is in the prompt. That is the failure mode of a large k: you have reinvented pasting the whole thing in. [SET top k BACK TO 2, CLICK 'what is the capital of France'] And here is the one I really want you to see. That question is not answerable from this document at all — there is nothing about France anywhere in it. Look at the top score: zero point one three six, against zero point six seven for the annual leave question. These are real embeddings, so that collapse is real. And the panel underneath flags it: nothing here really answers that. But now look at the chunk list. You still got two results back. Highlighted. Neatly ranked. Ready to send. That is the point from earlier, made concrete. A retriever always returns k results. It has no concept of "I do not know". If you hand those two chunks to a language model with no score threshold, it will do its best with irrelevant text — and that is where a large share of RAG hallucinations actually come from.

17 · The re-run trap

Run it twice, pay twice

from_documents ADDS, it does not replace Run the script twice and you have 1,094 vectors, half of them duplicates, and you paid to embed everything twice. While you are experimenting, delete db_chroma/ before each run.

The demo script clears the folder by default for exactly this reason — --append opts back in to the mistake so you can see it happen.

One trap before we move on, and it is one I have watched people fall into more than once. From documents adds to an existing collection. It does not replace it. So if you run your ingestion script twice — which you will, because you will tweak the chunk size and re-run — you end up with one thousand and ninety-four vectors, half of them exact duplicates. And you paid to embed all of them twice. The symptom is subtle rather than dramatic. Retrieval still works. But your top-k results start coming back as pairs of identical chunks, so asking for five results gets you two or three distinct pieces of information instead of five. Your context is half wasted and the quality quietly drops. While you are experimenting, delete the database folder before each run. In production you would use a proper upsert with stable document IDs, so re-ingesting updates a chunk instead of duplicating it.

Part five

The mistake that breaks
most first builds

Nothing crashes. There is no error message.

18 · The consistency rule

The consistency rule

Use the same embedding model and the same dimension count for your documents and for your queries. Every time. No exceptions.

Documents embedded in January

text-embedding-3-large
3,072 dimensions

Queries embedded in March

text-embedding-3-small
1,536 dimensions

Think of them as separate languages A vector written by one model means nothing to another. The two systems cannot understand each other, and neither one will tell you.

The consistency rule. Use the same embedding model and the same dimension count for your documents and for your queries. Every time. No exceptions. The scenario that produces the failure is completely ordinary. You embed your documents in January with one model. Two months later you come back, you are writing the query side, and you reach for a model — maybe a cheaper one, maybe you just do not remember which you used. Different model. Now your documents and your questions are in two different systems. The way to hold this in your head: think of embedding models as separate languages. A vector written by one model means nothing to another. The numbers are the same shape, they are in the same range, they look completely normal — and they encode meaning in a totally different arrangement. The two systems cannot understand each other. And neither one will tell you. Let's break it and watch.

18 · Break it on purpose

Same store. Same question. Nothing crashes.

The same 547 chunks · only the query model changed

question: 'Who founded SpaceX?' (answer lives in spacex.txt) RIGHT model text-embedding-3-small 0.662 spacex.txt In early 2002, Elon Musk started to look... 0.628 spacex.txt SpaceX was founded in 2002 by Elon Musk... 0.627 spacex.txt Space Exploration Technologies Corp., ... WRONG model text-embedding-3-large@1536 0.064 spacex.txt === Starbase, Texas === SpaceX manufact... 0.055 spacex.txt In December 2022, the U.S. Federal Comm... 0.052 microsoft.txt Microsoft became the third publicly ... ← WRONG FILE

Scores collapse from 0.66 to 0.06, and a Microsoft chunk is returned for a question about SpaceX.

So let us break it deliberately, and see what that looks like. This builds a store the right way — documents embedded with text-embedding-3-small — and then queries it the wrong way. One detail so you know it is a fair test. The wrong model is text-embedding-3-large, but asked for fifteen hundred and thirty-six dimensions instead of its default three thousand. So the vector is exactly the same shape as the ones in the database. Chroma has no dimension mismatch to complain about. It has no way whatsoever to know it is being handed a different language. Top block, the right model. Zero point six six, zero point six three, zero point six three. All from spacex dot txt. Bottom block. Same database. Same question. Same number of results. The only thing that changed is which model embedded the question. Look at the scores. Zero point zero six four. Zero point zero five five. Zero point zero five two. They have collapsed by a factor of ten — the retriever is finding nothing it considers a good match, because every stored vector is effectively noise to it now. And look at the third result. Microsoft dot txt, returned as a top-three answer to "who founded SpaceX". Now notice what did not happen. Nothing crashed. No exception. No warning. No log line. The retriever returned exactly three results, ranked, formatted identically to the correct run. If you were not printing the source filenames — and most people do not, in a first build — this looks completely healthy. And then an LLM takes those three chunks and writes a fluent, confident, sourced-looking, wrong answer. That is why this is the mistake that breaks most first builds. Not because it is subtle to fix, but because it is invisible until someone checks an answer by hand.

18 · Interactive

Try it: flip the model, break the system

Same store · same question · different query model real precomputed scores

0.00

top score

—

model used for the question

errors raised

And finally, the failure, as a switch you can flip. The documents on the left were embedded once, correctly, with text-embedding-3-small. The only thing this toggle changes is which model embeds the question. [CLICK 'SAME model'] Same model. Top score zero point five five, and the right answer — Q1 revenue reached four point two million — is clearly first. Healthy. [CLICK 'DIFFERENT model'] Different model. Same store, same question, same six facts. Top score, zero point zero two six. The signal has collapsed by a factor of twenty. The ordering is now essentially arbitrary — on six facts it happens to keep the right one near the top, which is exactly why this is so dangerous on a small test set. In demo eight, on five hundred and forty-seven real chunks, this same flip returned a Microsoft paragraph for a question about SpaceX. And now look at the bottom-right number, which is the whole point of this slide. Errors raised: zero. Not one exception. Not one warning. The retriever returned six results, ranked, formatted identically. Every piece of monitoring you have says the system is fine. [FLIP BACK AND FORTH A FEW TIMES] This is what a silent failure looks like. The difference between a working RAG system and a broken one, from the outside, is a number getting smaller. Nothing else changes. That is why the rule is absolute. One embedding model, one dimension count, everywhere.

18 · In practice

What this means in practice

Choose your embedding model before you ingest a single document.
Choose your dimension count at the same time, and write it down.
If you switch model later, you re-embed the entire corpus. There is no partial migration.
Changing dimensions within the same model breaks it too.

A cheap defence Store the model name and dimension count in the collection metadata when you build the store, and assert on it when you query. Ten lines of code that turn a silent failure into a loud one.

So what does that mean in practice? Four rules. One. Choose your embedding model before you ingest a single document. This is an architectural decision, not an implementation detail you get to defer. Two. Choose your dimension count at the same time, and write it down. In the README, in a config file, in a comment at the top of the script. Somewhere your future self will actually look. Three. If you switch model later, you re-embed the entire corpus. There is no partial migration. You cannot have half your documents in the new model and half in the old, because they cannot be compared. For a large corpus that is a real cost, which is why rule one matters. Four, and this one catches people who think they are being careful: changing dimensions within the same model breaks it too. Text-embedding-3-large at three thousand and text-embedding-3-large at fifteen hundred are not compatible with each other. Same model, different language. And then a suggestion that is not in the lesson but which I would put in any real build. Store the model name and the dimension count in the collection metadata when you create the store, and assert on them when you query. That is about ten lines of code, and it converts this entire category of silent failure into a loud one that fails on the first query. Given what we just watched, that is an extremely good trade.

19 · When it does not run

When it does not run

Error	Cause and fix
`OpenAIError: api_key must be set`	No `.env`, wrong variable name, or `load_dotenv()` missing. The name must be exactly `OPENAI_API_KEY`.
`RateLimitError: quota exceeded`	The key works but the account has no credit. Add funds in Billing. This is not a rate limit, despite the name.
`ModuleNotFoundError`	The virtual environment is not active, or you installed into a different one. Check for `(venv)` in your prompt.
`No .txt files found`	You are running the script from a different directory than the one holding `docs/`.
`Created a chunk of size N, longer than 800`	A warning, not an error. A single paragraph exceeded the target. Expected behaviour.

The five things that go wrong, and what each one actually means. OpenAI error, api key must be set. That is no dot env file, or the wrong variable name, or you forgot to call load dot env. The name has to be exactly OPENAI underscore API underscore KEY. Rate limit error, quota exceeded. This one is badly named and sends people down the wrong path. It is not a rate limit. Your key works fine — the account simply has no credit. Add funds in Billing. If you are sitting there adding retry logic and backoff, you are solving a problem you do not have. Module not found error. Your virtual environment is not active, or you installed into a different one. Check for the venv prefix in your prompt. This is the one that comes back to bite people who open a new terminal tab and forget to re-activate. No dot txt files found. You are running the script from a different directory than the one holding docs. Relative paths are relative to where you launched python, not to where the file lives. And created a chunk of size N longer than eight hundred. That is a warning, not an error. A single paragraph exceeded the target. Expected behaviour, as we saw. Nothing to fix.

19 · Check yourself

Check yourself

Why does a 2 million token context window not remove the need for RAG? Two reasons.
You have 4 million tokens of documents and a chunk size of 500. How many chunks, and how many vectors?
A retriever returns 10 chunks and 4 are irrelevant. Is the system broken?
What exactly is in the prompt that reaches the LLM?
Which attribute of a Document survives chunking?
chunk_size=800 counts what unit?
You leave out persist_directory. What happens when the script ends?
You embedded documents with model A and queries with model B. What is the symptom?

Before you move on, check yourself against these. If you are watching a recording of this, pause here — genuinely try to answer them out loud before reading on. One. Why does a two million token window not remove the need for RAG? Two reasons: scale, because real corpora are orders of magnitude larger; and cost and quality, because sending irrelevant context is expensive, slow, and produces worse answers. Two. Four million tokens at a chunk size of five hundred. Eight thousand chunks, and eight thousand vectors. One vector per chunk, always. Three. Ten chunks back, four irrelevant. Not broken. A retriever always returns k results whether or not they are any good. Four. What is in the prompt? The user's question and the original English text of the top chunks. Never the vectors. Five. Which Document attribute survives chunking? Metadata. Every chunk inherits its parent's. Six. Chunk size counts characters, not tokens. Seven. Leave out persist directory and Chroma runs in memory — everything vanishes when the script ends and you pay to embed again. Eight. Documents with model A, queries with model B. The symptom is that there is no symptom. No error, collapsed scores, and confidently wrong chunks.

19 · Summary

The whole session in ten lines

Models have a token limit. Real document stores are millions of times larger.
So you retrieve the relevant pieces instead of sending everything.
A RAG system is two pipelines. Ingestion runs once; retrieval runs per question.
Ingestion: cut documents into chunks, embed each chunk, store vectors with their text.
An embedding is a fixed-length list of numbers where similar meaning gives similar numbers.
A vector database stores those numbers and searches them fast.
Retrieval: embed the question the same way, rank by closeness, take the top k.
Send the model the question plus the original text. Never the vectors.
Set chunk_overlap so sentences on the seam survive. Set persist_directory or you lose the work.Overlap your chunks so sentences on the seam survive. Tell the database where to save, or you lose the work.
One embedding model, one dimension count, everywhere. Breaking this fails silently.

The whole session in ten lines. This is your revision card. Models have a token limit, and real document stores are millions of times larger. So you retrieve the relevant pieces instead of sending everything. A RAG system is two pipelines. Ingestion runs once. Retrieval runs per question. Ingestion: cut documents into chunks, embed each chunk, store the vectors with their text. An embedding is a fixed-length list of numbers where similar meaning gives similar numbers. A vector database stores those numbers and searches them fast. Retrieval: embed the question the same way, rank stored vectors by closeness, take the top k. Send the model the question plus the original text of those chunks. Never the vectors. Set chunk overlap so sentences on the seam survive, and set persist directory or you lose the work. And the last one, which is the one that will actually cost you an afternoon: one embedding model, one dimension count, everywhere. Breaking this fails silently.

That is Retrieval Augmented Generation

You take a question, embed it with that same model, pull the closest chunks out of db_chroma/, and hand them to an LLM to answer.

Two pipelines. Chunk, embed, store the text with the numbers. Ask with the same model, rank by closeness, send the paragraphs — never the vectors.

THE RESULT · 547 vectors on disk, searchable, with sources
LOCKED IN · text-embedding-3-small at 1,536 dimensions
BRING · a set of documents you actually want to ask questions about
YOU CAN NOW · draw both pipelines and say what each box is for
YOU KNOW THE TRAP · one embedding model, one dimension count, everywhere
TO GO FURTHER · the playgrounds, the code, and the long version
ALL OF IT · ragverse.diy

Retrieval AugmentedGeneration

What you will be able to do

Why RAG exists

Several hundred internal documents

Retrieval Augmented Generation

Where you have met one — and why

Models do not read words

Everything has to fit inside one window

Read the scale carefully

Try it: when do you actually need RAG?

You also pay per token

Send everything

Send the right five paragraphs

The shape of the system

Two pipelines

A chunk is a slice, not a section

This is not an LLM

Similar meaning, similar numbers

Does that actually hold?

Try it: the neighbourhood map

The ones you will meet first

The vector database stores both

Retrieval, in principle

The question takes the same road

You choose how many come back

Vectors find. Text is what gets sent.

Sent to the model

Not sent

What actually happens to your documents

You are here

Five articles

A document is its text, plus where it came from

Two things worth knowing

800 is a target, not a cap

What overlap actually does

Try it: watch the chunks re-cut

Every chunk becomes numbers, text and a source

Ask it who founded SpaceX

This is what reaches the model

Try it: ask, and watch the prompt build

Run it twice, pay twice

The mistake that breaksmost first builds

The consistency rule

Documents embedded in January

Queries embedded in March

Same store. Same question. Nothing crashes.

Try it: flip the model, break the system

What this means in practice

When it does not run

Check yourself

The whole session in ten lines

That is Retrieval Augmented Generation

Retrieval Augmented
Generation

The mistake that breaks
most first builds