RAG, end to end.

Welcome. Before anything else, a quick word on who is talking.

I am Wil. I have been working in AI for about a decade, which means I started well before GPTs made any of this fashionable — back when you built your own models and your own data pipelines because there was nothing to call. More recently I have been working on incentivised data training structures with NEAR AI: the mechanisms that reward people for contributing the data and the compute that models are actually trained on. And running through all of it, I have built a lot of retrieval and ingestion systems, on a lot of different stacks.

I mention that for one reason. This is not a lesson I read and repeated. It is the shape of a system I have built more than once — and every mistake in it is a mistake I have made.

So: we are going to build a Retrieval Augmented Generation system, and understand every piece of it.

Here is the one-line version of what RAG is for. How do you give a language model access to a hundred thousand documents, without ever putting a hundred thousand documents into the prompt? That is the problem. Everything after it is mechanics.

One practical thing before we start. Everything is at ragverse dot d i y — these slides, five interactive playgrounds you can drive yourself, and every line of the code. Open it in another tab now if you want to follow along. The address stays in the corner of every slide, so you do not need to write it down.

Let's start with why the problem exists at all.

These are the outcomes. Six of them.

By the end you will be able to explain why a bigger context window does not remove the need for RAG — that is the objection everyone raises first, and it has two answers.

You will be able to define the vocabulary: token, context window, chunk, embedding, dimension, vector database, retriever, top k. These words get used loosely and that is where confusion comes from, so we will pin each one down as it appears.

You will be able to draw both halves of a RAG system from memory. There are two halves and people routinely collapse them into one, which is where the mental model breaks.

You will be able to look at a question and predict which chunks come back, and say why.

You will have built a working ingestion pipeline — real code, on your machine, about sixty lines.

And last, you will know the one mistake that silently breaks most first builds. Silently is the important word there. We will break it on purpose at the end so you know what it looks like.

Picture a company with several hundred internal documents. Policy guidelines, technical specifications, customer support logs, contracts. Ordinary business documents.

Somebody asks a question, and exactly one of those documents contains the answer. The obvious move — and this is genuinely what everyone tries first — is to paste all of them into the model and ask the question.

That does not work. And the reason it does not work is the whole reason this lesson exists. So let me give you the definition to hold on to, and then we will take apart why the obvious approach fails.

Here is the definition. Retrieval Augmented Generation is a language model combined with a retrieval system. The retrieval system searches external sources — documents, databases, knowledge bases — and it pulls only the relevant pieces into the prompt, at the moment the model needs them.

Read that last part again, because it is the part people skip. At the moment the model needs them. Nothing is loaded in advance into the model. Nothing is trained into the model. The relevant text is fetched, per question, and placed in the prompt.

So the model does not get everything. It gets the right few pages.

That is the entire idea. I mean that literally — everything else in this session is mechanics for how you find the right few pages quickly. If you leave with only one sentence, leave with that one.

Before any mechanics, the question underneath all of this: why does anyone build one?

Start with where you have already met one, because you have. If you have used a support chatbot that answered your question and then linked you to the exact help article it got that from — that was retrieval. If you have watched a model search the web and cite what it found — retrieval. If you have asked a coding assistant something about your own repository and got a real answer rather than a plausible one — retrieval. Every company with an internal Q&A bot over its own policies is running one of these.

Now the reasons teams build them. There are three, and they are worth separating.

The first is your data. Whatever model you are using was not trained on your contracts, your runbooks, or your support tickets. It cannot know them. Ask it anyway and it will cheerfully invent something that sounds right.

The second is today's data. A policy changed this morning. Somebody shipped a new product page an hour ago. You are not going to retrain a model because a policy changed — you drop the document in, and the next question picks it up.

The third is receipts, and this is the one people underrate. The retriever knows exactly which chunks it used to answer, so the answer can point back at them. In most organisations an answer nobody can check is worth close to nothing. Being able to click through to the paragraph it came from is often the entire reason the project got approved.

And then the question I get every single time: why not just fine-tune the model on our documents?

Fine-tuning teaches a model a style. A format, a voice, a way of reasoning about a domain. It is a very expensive way to teach it a fact, and a hopeless way to teach it a fact that changes on Tuesday. It also gives you nothing to cite. Fine-tuning changes how a model speaks. Retrieval changes what it knows, and lets it show its working.

A word you will need: the token.

A token is the unit of text a model processes. Sometimes it is a whole word. Often it is a fragment of one. Models do not read letters, and they do not read words — they read tokens. This matters because every limit and every price you will ever meet is denominated in tokens, not words.

Here is what a real tokenizer does with a five-word sentence.

Look at that. "Retrieval augmented generation is powerful" is five words. The model counts seven tokens. And look at how it split: "Retrieval" — a fairly ordinary English word — got broken into three pieces, R-e-t, r-i-e, v-a-l. Whereas "augmented", "generation" and "powerful" each survived as a single token.

That is the rule in action. Common words are one token. Rarer or longer words get split into several. And those underscores are spaces — spaces travel with the token that follows them.

The rule of thumb worth memorising: one token is roughly three quarters of an English word. So a thousand words is roughly thirteen hundred tokens.

The next word you will need: the context window.

The context window is the total number of tokens a model can hold in a single request. And the critical detail is that everything counts toward it — everything you send in, and everything the model writes back out. They share one budget. Past that limit, the information is simply not there. Not summarised, not compressed. Not there.

Right now, in 2026, the major models have converged on one million tokens. GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8, Claude Sonnet 5 — they all sit at a million. A few advertise ten million, though I'd treat that with some suspicion: no published benchmark shows answer quality holding anywhere near the top of those windows. Advertised capacity and usable capacity are not the same thing.

Still — a million tokens is a genuinely enormous number. It is several very long books.

I want to flag something about that figure: verify it before you quote it. These numbers move every few months, and a slide deck is exactly the kind of place a stale number goes to live forever.

So — two million tokens. That sounds like it should be plenty. Let's see how it compares to how much text an actual company holds.

This is the same demo still running — the second half of demo one.

Now, this is a logarithmic scale, and I need you to read it carefully, because logarithmic scales are quietly deceptive. Each step to the right is ten times larger than the last, not one step larger. The bars look comparable. The numbers are not.

Start at the top. One chunk — that is a single retrievable piece of text, and we will define it properly in a moment — is about a thousand tokens.

The five Wikipedia articles we are about to ingest come to seventy-two thousand tokens. That is our whole corpus for today, and notice — it comfortably fits inside a frontier model. For five documents you genuinely do not need RAG. I want to be honest about that rather than pretend otherwise.

A frontier model window: one million. The largest advertised: ten million.

Now watch what happens. A mid-sized company with one terabyte of documents: two hundred and fifty billion tokens. That is two hundred and fifty thousand times larger than the model window.

An enterprise archive at one petabyte: two hundred and fifty trillion.

So here is the thing to take away. That gap is not a gap you close by waiting for bigger models. If context windows got a thousand times bigger tomorrow — which they will not — you would still be short of the mid-sized company by a factor of two hundred and fifty. This is a structural problem, not a temporary one.

Let's make that scale argument something you can feel rather than just read, because the numbers are so large they stop meaning anything.

Drag the slider. It runs from one megabyte of text up to one petabyte.

[DRAG TO ~1 MB] At a megabyte we are at about two hundred and fifty thousand tokens. That fits in a frontier window with room to spare, and the panel says so — you do not need RAG for this. Be honest about that. If your entire corpus is a handful of documents, retrieval is overhead, not architecture.

[DRAG TO ~1 GB] One gigabyte. Two hundred and fifty million tokens. Now we need over a hundred context windows. There is no prompt you can write that holds this.

[DRAG TO 1 TB] And a terabyte — a mid-sized company — needs a hundred and twenty-five thousand full context windows.

Watch the cost figure while you drag, because that is the one people do not think about. Embedding a terabyte once costs about five thousand dollars. That is a one-off, not per question — but it is a real number, and it is why you choose your embedding model carefully before you start rather than after.

The threshold where RAG stops being optional is lower than people expect. Somewhere around a few megabytes, this stops being a choice.

There is a second reason, and it matters just as much as the first, but people forget it because the first one is so dramatic.

Even when everything does fit — even in the case where your documents are small enough — sending all of it is still the wrong move.

Three reasons. One, you pay per token, so you are paying for every irrelevant word. Two, it is slow; latency scales with how much you send. And three — this is the one that surprises people — you get worse answers.

That third point is counterintuitive so let me be explicit about it. If you bury the one relevant paragraph inside five hundred thousand tokens of unrelated material, the model has a harder job finding it than if you had simply handed it the paragraph. More context is not more helpful. Relevant context is helpful.

So: sending five hundred thousand tokens of irrelevant context to answer one question is expensive, slow, and worse. Sending the five right paragraphs is cheap, fast, and better.

That is the case for RAG, complete. Now let's look at how it is actually built.

Here it is. The whole system in one picture.

The top row is the ingestion pipeline, and the thing to understand about it is that it runs once, ahead of time, before anybody asks anything. Source documents go in. They get chunked — cut into small pieces. Each piece goes through an embedding model, which turns it into a vector, a list of numbers. Those vectors get stored in a vector database. Done. That pipeline does not run again until your documents change.

The bottom row is the retrieval pipeline, and it runs every single time someone asks a question. The question comes in. It goes through an embedding model and becomes a vector. A retriever compares that vector against everything in the database and ranks by closeness. The top matching chunks come back. And those chunks, plus the question, go to the LLM, which writes the answer.

Now look at the two orange boxes. The embedding model appears in both rows. That is not me being lazy with the drawing. It is the same model, and it has to be the same model. If those two boxes ever contain different models, the entire system fails — and it fails silently, with no error message. We are going to break that rule deliberately in the last demo so you know exactly what it looks like.

Keep this diagram. Every RAG system you will ever build, from a weekend project to production, is this diagram with more engineering around each box.

Next word: the chunk.

Chunking is breaking large documents into small pieces. You choose the size. If you set it to a thousand tokens, then ten million tokens of documents becomes ten thousand chunks. Simple arithmetic.

Now, the word "chunk" is doing some work here and I want to be precise. A chunk is a slice, not a section. Chunks are cut by size, not by meaning. They do not respect chapter boundaries, headings, or the structure of an argument. A naive splitter will cut straight through the middle of a sentence, or a table, without hesitating. You will see exactly that when we get to the chunking step.

So why cut at all? Why not keep whole documents? Because the chunk is your unit of retrieval — it is the smallest thing the system is able to hand back. If your chunks are whole Wikipedia articles, then a question about one sentence returns the entire article, and you are back to stuffing the context window with mostly-irrelevant text. Which is the problem we started with.

Chunk size is therefore a real design decision, and there are techniques for cutting more intelligently. Those come later in the track. For today we cut simply, and I will show you exactly what that costs.

Next box in the diagram: the embedding model. And the first thing to say, because it trips everybody up, is that this is not an LLM. Different model, different job. It does not generate text. It does not chat. You give it text, and it gives you back a list of numbers. That is all it does.

Look at the three rows on screen. One word in. An eight-word sentence in. A three-hundred-and-thirty-seven-word paragraph in. And every single time, what came back was fifteen hundred and thirty-six numbers. Not more for the paragraph. Not fewer for the single word. Always exactly fifteen thirty-six.

Underneath you can see the first few actual numbers. They are small, they are signed, and — this is worth saying plainly — no individual number there means anything you can name. Nobody knows what dimension four hundred and twelve represents. That is fine. It is not how they are used.

The property that matters is the fixed length. One word in or nine hundred words in, the vector is the same size. That is the property the entire system is built on: any two pieces of text, however different in length, become two lists of the same size — and two lists of the same size can be compared with simple arithmetic.

That is the trick. Everything downstream depends on it.

So what is an embedding, actually? This is the part people get stuck on, so we will slow down here.

A vector embedding is a list of numbers that stands in for a piece of text. Each number in that list is called a dimension. Two more words for the list — and dimension just means one position in it.

And here is the sentence that matters more than anything else in this session: text with similar meaning produces similar numbers.

It is worth being concrete about why that is the whole ballgame. Somebody types "my card got declined". The help article that answers them says "payment failures". Not one word in common. A keyword search — the thing every support site had for twenty years — returns nothing at all, and the customer gives up. An embedding model puts those two pieces of text next to each other, because it was trained on how words are used rather than how they are spelled. Matching on meaning instead of spelling is the entire thing you are buying here. That is the property. That is what an embedding model is trained to do.

Now, this table is a teaching device, and it is a fake. I have made up three dimensions and labelled them size, domesticated, and sound. Real embeddings do not work like this — the dimensions are not labelled, and nobody knows what any individual one means. But the fake is useful for building intuition, so read it column by column.

Cat and kitten sit almost on top of each other on every dimension — thirty-four and thirty-three on size, both eight on domestication, seven point five and seven point one on sound. Because a kitten is a young cat. Dog shares the domestication value but sits further out on size. And elephant is far away from all three on everything — two hundred and ten on size against thirty-four.

That is the intuition. Now let's check whether it survives contact with a real embedding model, because I do not want you taking a made-up table on faith.

So does that claim actually hold? I did not want to assert it and move on, so here it is measured.

This takes eight ordinary words, embeds each one with a real OpenAI model, and measures how close every pair is. One point zero would be identical.

Top of the list: cat to dog, zero point six oh three. Then cat to kitten at zero point five seven. Then a clear drop — coffee, tea, apple, elephant, mango, all down in the threes and twos.

Now, I want to point at something honest here, because it contradicts the tidy table on the previous slide. That table said kitten should be closest to cat. The real model puts dog first, with kitten second. So the toy intuition was directionally right and specifically wrong.

Why? Because bare single words are a weak signal. "Cat" and "dog" co-occur in text constantly — they are the two canonical pets — whereas "kitten" is a narrower word. Real systems almost never embed single words; they embed whole paragraphs, where there is far more meaning to work with. So do not over-read the ordering inside a cluster.

But look at the bottom two lines, because that is where the claim really holds. Domestic animals: average similarity inside that group is zero point five three, and to everything outside it, zero point three two. Drinks: zero point six one inside, zero point three one outside. Every group is meaningfully tighter inside than out.

That is a neighbourhood. And the crucial part — nobody labelled any of this. No human told the model that coffee and tea are related. It fell out of arithmetic over fifteen hundred numbers.

Now you can play with it yourself. These are real vectors — genuine fifteen-hundred-dimension OpenAI embeddings, computed ahead of time and shipped with the slide so it works without an API key.

Click any word and it becomes the probe. Everything re-ranks against it.

On the left, similarity as bars, coloured by category — orange for animals, green for fruit, purple for drinks. On the right, a two-dimensional projection of all fifteen hundred and thirty-six dimensions. That projection throws away most of the information, so treat it as a sketch rather than the truth — but the clustering it shows is real.

[CLICK 'coffee'] Look — tea comes straight to the top. Water is up there too. The animals fall away.

[CLICK 'puppy'] Puppy. Dog first, obviously. Then kitten, then cat. It has learned that puppy-dog and kitten-cat are the same relationship, without anybody telling it what a young animal is.

[CLICK 'lion'] And here is a nice one. Lion sits near cat and elephant but away from dog — it has picked up something about wild versus domestic that we never labelled.

Try a few. The thing I want you to notice is that every ordering you see came out of arithmetic on numbers, and every one of them is defensible. That is what "similar meaning gives similar numbers" actually looks like in practice.

There are many embedding models. These are the ones you will meet first.

Text-embedding-3-small from OpenAI, fifteen hundred and thirty-six dimensions. Cheap, and fine for most projects. That is the one we are using today, and I want you to notice that I am telling you the model and the dimension count in the same breath. That habit will make sense shortly.

Text-embedding-3-large, three thousand and seventy-two dimensions. Better quality, costs more.

Then Voyage, which is strong on technical and code text — worth knowing if you are indexing a codebase. Cohere, good multilingual support. And Mistral, which has open weights if you need to run it yourself.

One genuinely useful thing at the bottom. Most of these models let you request fewer dimensions than the default. If you ask a three-thousand-dimension model for five hundred and twelve dimensions, you cut your storage by a factor of six and typically lose very little accuracy. That is a real lever in production and most people do not know it exists.

Same caveat as the context windows: check current pricing and dimension options in the provider documentation before you commit. These move.

There is a trade-off underneath all of this. More dimensions capture more meaning, and also cost more to compute and more to store. Most production systems reduce dimensions deliberately.

Last box in the ingestion row: the vector database.

A vector database stores embeddings and finds the closest ones to a given vector, fast. That is its specialty — not storing things, but searching them by proximity across a huge number of dimensions.

But here is the detail people miss, and it is the reason I put this table up. Crucially, it stores the original text alongside each vector. Look at the columns: an ID, the vector, the original text, and the source file. One row per chunk.

The vector is how it gets found. The original text is what gets used. If you store only the numbers, you have a beautiful search index and absolutely nothing to send to a language model — because you cannot turn a vector back into English. It is a one-way trip.

The fourth column, source, is metadata. That is what lets you tell a user "this answer came from policy dot pdf, page four". We will watch that column survive all the way through the pipeline.

For options: Pinecone, Weaviate, Chroma, and Qdrant are purpose-built vector databases. We are using Chroma today because it runs locally, on disk, with no account and no server. FAISS is a library from Meta rather than a hosted service. And if you already run Postgres, the pgvector extension turns it into a vector database, which is often the right answer in a real company.

Here is the whole retrieval pipeline in four steps.

Step one. The question goes through the same embedding model the documents went through. Same model. I am going to keep repeating that until it is annoying, because it is the thing that breaks.

Step two. It comes out as a vector of the same length — fifteen hundred and thirty-six numbers, exactly like every chunk in the database.

Step three. The retriever compares that one vector against every stored vector. Conceptually every one; in practice the database uses an index to avoid a brute-force scan, but the effect is the same.

Step four. It ranks them by closeness and returns the top k.

Look at the example. This one is drawn, not run — it says so at the top — because I want a tidy invented company here rather than five Wikipedia articles. The point is easier to see.

The question is "what were our sales in the first quarter". It becomes a vector. And then every chunk gets a score. Q1 revenue reached four point two million: zero point nine one. Quarterly sales by region: zero point eight eight. Revenue targets for the first quarter: zero point eight four. Those three go to the model.

And then look at what did not make it. Annual headcount summary at zero point four two — related to business, not to the question. Refunds at zero point three one. Guest wifi at zero point one nine, which is about as unrelated as it gets.

Notice something about that top result. The question says "sales". The winning chunk says "revenue". Those are different words. A keyword search for "sales" would have missed it entirely. That is what embeddings buy you — matching on meaning rather than on spelling.

How closeness is actually calculated is a later topic. For now: it is distance between two points, measured with arithmetic.

The number of chunks that come back is called top k. That is the last of the words worth collecting. You choose it. Ask for the top five and you get five results.

And now the important half of that sentence: whether or not all five are any good.

A retriever always returns something. It has no concept of "there is no good match here". If your entire corpus contains nothing relevant to the question, you still get k results back, neatly ranked, looking perfectly respectable. They are simply the least-bad of a bad set.

This catches people out constantly. Somebody builds their first RAG system, asks it a question the documents genuinely do not answer, gets five confident-looking chunks back, and concludes the retriever is broken. It is not broken. That is the expected shape of the output.

So if a retriever returns ten chunks and four are irrelevant — that is normal. Filtering on a score threshold, or re-ranking, is how you handle it, and that is a later topic in the track. But knowing that it happens is not a later topic. That is today.

This is the step everyone misreads, so I am giving it its own slide.

There is a very common mental model in which the vectors somehow go to the language model — as if the LLM understands embeddings, or as if the numbers are a compressed form the model can read. That is wrong, and it is worth killing off explicitly.

What is sent to the model: the user's question, and the original English text of the top chunks. That is it.

What is not sent: the vectors. Anything numeric at all.

Vectors are only used for finding. Their job is over the instant the matching is done. After retrieval you are back in plain English, and the prompt that reaches the model is the question plus a few paragraphs of perfectly ordinary text — the kind of thing you could have pasted in by hand.

Once that clicks, a lot of RAG stops being mysterious. The clever part is the search. The prompt at the end is boring, and it is meant to be. We are going to print that exact prompt to the screen later so you can read every character of it.

Right. That is all the theory. Let's build it.

Orienting you before we start typing. This is the same two-row diagram, with the concrete numbers for today filled in.

Five text files in a docs folder. They become five hundred and forty-seven chunks. Those go through OpenAI's embedding model and become five hundred and forty-seven vectors, which land in a Chroma database on disk.

The bottom row — question, embed, retrieve, build a prompt, answer — is the next lesson. We will touch it briefly at the end today just to prove the database works, but building it properly is next time.

And again: ingestion runs once. When we finish, you will not run this code again unless your documents change. That is worth internalising because it shapes how you think about cost. The expensive step happens once, up front, and every question after that is cheap.

You need documents. For this walkthrough I used the Wikipedia articles for Google, Microsoft, Nvidia, SpaceX and Tesla, saved as plain text — about three hundred and fifty thousand characters across five files.

Any five text files work. Company articles are convenient for teaching because they are long, factual, and full of specific names and numbers you can test the retriever against. When you ask "who founded SpaceX" you know what the right answer is, so you can tell immediately whether retrieval worked.

One detail worth mentioning, because it is the kind of thing that quietly ruins a pipeline. Wikipedia's plain-text export separates paragraphs with a single newline. The splitter we are about to use splits on blank lines. So the fetch script normalises every paragraph onto its own blank-line-separated block. Skip that step and the splitter finds almost no break points and produces eight-thousand-character chunks, which are useless units of retrieval.

That is a real data-cleaning step, and it is invisible until you look at your chunk sizes.

Here is what comes back. Five files in, five Document objects out.

That Document object shows up everywhere from here on, so learn its two attributes now — there are only two and they are both simple.

Dot page underscore content is the entire text of the file as one long string. Not a list of lines, not a stream. One string — sixty-eight thousand characters for Google, ninety-four thousand for Tesla.

Dot metadata is a dictionary. Right now it holds one key, source, pointing at the file path — we put it there ourselves, which is worth noticing, because it means you can put anything you like in it. You can add your own keys later — page numbers, authors, dates, permissions — and that is how real systems do access control and citation.

And here is the property that makes metadata worth caring about: metadata survives everything. When a document gets chunked in a moment, every single chunk inherits its parent's metadata. That is how a RAG system can tell a user which file an answer came from, even though the thing it retrieved was a four-hundred-character fragment.

Two things worth knowing.

First, that call to sorted is not decoration. DirectoryLoader walked the directory in whatever order the filesystem handed back — which meant index zero could be a different file on your machine than on mine, and the chunk order changed with it. For a lesson where we compare numbers, that is genuinely annoying. Sorting makes the run reproducible.

But the underlying advice survives either way: never write code that assumes index zero is a particular file. If you need a specific document, filter on the metadata.

Second, text is the easy case. I have just told you that you do not need a loader package, and for dot-txt files that is true. PDFs, spreadsheets and HTML are a different story — those need a real parser, and that is absolutely worth a dependency. Reach for langchain-unstructured, or pypdf directly.

The important part is that what comes out the other side is the same thing: Document objects, with page content and metadata. So everything downstream of this box — chunking, embedding, storing, retrieving — does not change at all.

Five hundred and thirty-nine chunks, from five documents.

Now look at the spread, because this is where the mental model needs correcting. The smallest chunk is fifteen characters. The largest is one thousand seven hundred and nineteen. The average is six hundred and forty-three. We asked for eight hundred and got almost nothing that is actually eight hundred.

Seventy-nine of the five hundred and thirty-nine came out longer than the target.

Here is why. CharacterTextSplitter splits on the separator first — blank lines, in our case — and then merges the resulting pieces back together up to chunk size. What it will not do is break a paragraph in half to hit the number. So if a single paragraph is seventeen hundred characters, you get a seventeen-hundred-character chunk, and LangChain logs a warning for each one.

Those warnings are warnings, not errors. I have suppressed them in the demo and counted them instead, because otherwise seventy-nine lines scroll past and you cannot see anything else. But you will see them if you write this yourself, and now you know they are expected.

Chunk size is a target, not a hard cap. RecursiveCharacterTextSplitter handles this better — it tries a sequence of separators, falling back to smaller ones — and it is what you would reach for in a real project. We are staying on the simple one today so that the mechanics stay visible.

This is the part of demo five I actually care about, and it is shown on a short passage with a small chunk size so the whole thing fits on one screen. The mechanism is identical at eight hundred characters.

Top block, overlap zero. Read the seam between chunk zero and chunk one. Chunk zero ends with "its name is a tribute to the". Chunk one begins with "inventor Nikola Tesla". The sentence is cut clean in half.

Think about what that costs you. Neither half carries the whole fact. If someone asks "who is Tesla named after", chunk zero has the question's subject but not the answer, and chunk one has the answer but has lost the subject. Neither one is a good match. The fact is in your database and it is unreachable.

Bottom block, overlap forty. Now chunk one begins by repeating the tail of chunk zero — "Motors, its name is a tribute to the" — and then continues into "inventor Nikola Tesla". The sentence on the seam survives intact inside one chunk. It is retrievable again.

That is all overlap does. Each chunk repeats the last n characters of the one before it.

Rule of thumb: set overlap to roughly ten to twenty percent of chunk size. For eight hundred characters, one hundred is a reasonable default. You pay for that duplication in storage and in embedding cost — the same text gets embedded twice — and it is almost always worth it.

So we change the default to a hundred, and re-run.

This is the same two numbers you just saw in code, except now you can move them and watch what they cost you.

The document is the opening of the Tesla article — the same text demo five used.

[DRAG chunk_size DOWN TO ~150] Small chunks. Look at the count climbing and the average dropping. These are precise — each one is about a single thing — but read them. They have lost their surroundings. A chunk that says "in 2008 he was named chief executive officer" does not say who "he" is. Retrieve that on its own and it is useless.

[DRAG chunk_size UP TO ~1200] Now large chunks. Only a couple of them. Each one carries plenty of context — but now a question about the Roadster matches a chunk that is mostly about something else, and you are sending three times as many tokens to answer it. The match gets diluted.

[DRAG chunk_overlap UP TO ~100] Now watch the highlight. That green text at the start of each chunk is repeated from the one before it. Look at the "duplicated" figure climbing — that is text you are storing twice and paying to embed twice.

[SET overlap TO 0, THEN BACK TO 100] And that is the trade. At zero you pay nothing extra and sentences on the seam get cut in half. At a hundred you pay maybe fifteen percent more and they survive.

There is no correct answer on this slide. There is a shape of answer: chunks big enough to stand alone, small enough to be about one thing, with enough overlap that the seams do not eat your sentences.

Five hundred and forty-seven vectors stored. And a folder has appeared that was not there a moment ago.

Look at what is stored per chunk, because this is the vector database table from earlier, now real. The vector: fifteen hundred and thirty-six numbers. The original text: the opening of the Google article. And the metadata: source, docs slash google dot txt.

There is the metadata surviving, exactly as promised. It went in at the loader, came through the splitter untouched, and is now sitting in the database attached to a chunk.

Underneath, the folder structure. Chroma dot sqlite three holds the text, the metadata and the ids — it is an ordinary SQLite database, you can open it. And then a folder named after the collection's UUID holds the binary files: data level zero dot bin is the vectors themselves, and link lists dot bin is the search index, which is what makes proximity search fast rather than a brute-force scan of half a million numbers.

And the line to remember: the original text is not optional. Without it you have numbers and nothing to send an LLM. That is why this database stores both.

So does it work? Here is the store being asked a real question.

The question is "who founded SpaceX". The collection has five hundred and forty-seven vectors in it, covering five different companies.

And every single result comes from spacex dot txt. Not one Tesla chunk, not one Microsoft chunk — despite Elon Musk appearing prominently in the Tesla article, which is exactly the kind of near-miss that would trip a keyword search.

Look at the top hit: "In early 2002, Elon Musk started to look for staff for his company, soon to be named SpaceX." That answers the question. Score zero point six six two.

One thing to note about the scores. Zero point six six is the top match, and that might feel low if you were expecting something near one. It is not low. Cosine similarity between a short question and a long paragraph rarely goes above zero point seven, because they are different shapes of text. What matters is the gap between the top results and everything else, not the absolute number. Do not go hunting for a universal threshold — calibrate against your own corpus.

And here it is. The actual prompt. Every character of it, printed by the demo.

I promised earlier we would look at this, because it is the step everyone misreads. Read what is on that screen. An instruction: "answer the question using only the context below." Then the word CONTEXT, and then three paragraphs of completely ordinary English lifted straight out of the Wikipedia article. Then the word QUESTION, and the user's question.

That is it. That is the whole thing. Just under three thousand characters.

There is nothing numeric anywhere in it. No vectors, no embeddings, no similarity scores. The model has no idea a vector database was involved. As far as it is concerned, somebody pasted three paragraphs and asked a question.

Vectors are only used for finding. Their job finished the moment the matching was done, and then we went back to plain English.

I think this is the single most clarifying thing in the whole lesson. All the machinery — the chunking, the embedding, the fifteen hundred dimensions, the HNSW index — exists to produce those three paragraphs. The clever part is the search. The prompt at the end is boring, and it is supposed to be.

Last interactive one, and it puts the whole retrieval pipeline in your hands.

Small policy document, cut into chunks. Type a question, and every chunk gets scored against it. The highlighted ones are the ones that would be sent.

One honesty note before I start clicking. The four preset questions were embedded ahead of time with the real model, so those scores are genuine cosine similarity over fifteen hundred dimensions. If you type your own question, the page cannot embed it — that needs an API call — so it falls back to a rough local measure. The panel tells you which one you are looking at, and you should watch that label.

[CLICK 'laptop budget'] Watch the ranking jump. The equipment chunk goes to the top and the prompt underneath rebuilds itself in real time.

[DRAG top k FROM 2 TO 6] Now raise top k. More chunks get highlighted, and watch the "sent to model" figure climb — by six it reads a hundred percent. Every part of the document is in the prompt. That is the failure mode of a large k: you have reinvented pasting the whole thing in.

[SET top k BACK TO 2, CLICK 'what is the capital of France'] And here is the one I really want you to see. That question is not answerable from this document at all — there is nothing about France anywhere in it.

Look at the top score: zero point one three six, against zero point six seven for the annual leave question. These are real embeddings, so that collapse is real. And the panel underneath flags it: nothing here really answers that.

But now look at the chunk list. You still got two results back. Highlighted. Neatly ranked. Ready to send.

That is the point from earlier, made concrete. A retriever always returns k results. It has no concept of "I do not know". If you hand those two chunks to a language model with no score threshold, it will do its best with irrelevant text — and that is where a large share of RAG hallucinations actually come from.

One trap before we move on, and it is one I have watched people fall into more than once.

From documents adds to an existing collection. It does not replace it. So if you run your ingestion script twice — which you will, because you will tweak the chunk size and re-run — you end up with one thousand and ninety-four vectors, half of them exact duplicates. And you paid to embed all of them twice.

The symptom is subtle rather than dramatic. Retrieval still works. But your top-k results start coming back as pairs of identical chunks, so asking for five results gets you two or three distinct pieces of information instead of five. Your context is half wasted and the quality quietly drops.

While you are experimenting, delete the database folder before each run. In production you would use a proper upsert with stable document IDs, so re-ingesting updates a chunk instead of duplicating it.

The consistency rule. Use the same embedding model and the same dimension count for your documents and for your queries. Every time. No exceptions.

The scenario that produces the failure is completely ordinary. You embed your documents in January with one model. Two months later you come back, you are writing the query side, and you reach for a model — maybe a cheaper one, maybe you just do not remember which you used. Different model. Now your documents and your questions are in two different systems.

The way to hold this in your head: think of embedding models as separate languages. A vector written by one model means nothing to another. The numbers are the same shape, they are in the same range, they look completely normal — and they encode meaning in a totally different arrangement.

The two systems cannot understand each other. And neither one will tell you.

Let's break it and watch.

So let us break it deliberately, and see what that looks like.

This builds a store the right way — documents embedded with text-embedding-3-small — and then queries it the wrong way.

One detail so you know it is a fair test. The wrong model is text-embedding-3-large, but asked for fifteen hundred and thirty-six dimensions instead of its default three thousand. So the vector is exactly the same shape as the ones in the database. Chroma has no dimension mismatch to complain about. It has no way whatsoever to know it is being handed a different language.

Top block, the right model. Zero point six six, zero point six three, zero point six three. All from spacex dot txt.

Bottom block. Same database. Same question. Same number of results. The only thing that changed is which model embedded the question.

Look at the scores. Zero point zero six four. Zero point zero five five. Zero point zero five two. They have collapsed by a factor of ten — the retriever is finding nothing it considers a good match, because every stored vector is effectively noise to it now.

And look at the third result. Microsoft dot txt, returned as a top-three answer to "who founded SpaceX".

Now notice what did not happen. Nothing crashed. No exception. No warning. No log line. The retriever returned exactly three results, ranked, formatted identically to the correct run. If you were not printing the source filenames — and most people do not, in a first build — this looks completely healthy.

And then an LLM takes those three chunks and writes a fluent, confident, sourced-looking, wrong answer.

That is why this is the mistake that breaks most first builds. Not because it is subtle to fix, but because it is invisible until someone checks an answer by hand.

And finally, the failure, as a switch you can flip.

The documents on the left were embedded once, correctly, with text-embedding-3-small. The only thing this toggle changes is which model embeds the question.

[CLICK 'SAME model'] Same model. Top score zero point five five, and the right answer — Q1 revenue reached four point two million — is clearly first. Healthy.

[CLICK 'DIFFERENT model'] Different model. Same store, same question, same six facts.

Top score, zero point zero two six. The signal has collapsed by a factor of twenty. The ordering is now essentially arbitrary — on six facts it happens to keep the right one near the top, which is exactly why this is so dangerous on a small test set. In demo eight, on five hundred and forty-seven real chunks, this same flip returned a Microsoft paragraph for a question about SpaceX.

And now look at the bottom-right number, which is the whole point of this slide.

Errors raised: zero.

Not one exception. Not one warning. The retriever returned six results, ranked, formatted identically. Every piece of monitoring you have says the system is fine.

[FLIP BACK AND FORTH A FEW TIMES] This is what a silent failure looks like. The difference between a working RAG system and a broken one, from the outside, is a number getting smaller. Nothing else changes.

That is why the rule is absolute. One embedding model, one dimension count, everywhere.

So what does that mean in practice? Four rules.

One. Choose your embedding model before you ingest a single document. This is an architectural decision, not an implementation detail you get to defer.

Two. Choose your dimension count at the same time, and write it down. In the README, in a config file, in a comment at the top of the script. Somewhere your future self will actually look.

Three. If you switch model later, you re-embed the entire corpus. There is no partial migration. You cannot have half your documents in the new model and half in the old, because they cannot be compared. For a large corpus that is a real cost, which is why rule one matters.

Four, and this one catches people who think they are being careful: changing dimensions within the same model breaks it too. Text-embedding-3-large at three thousand and text-embedding-3-large at fifteen hundred are not compatible with each other. Same model, different language.

And then a suggestion that is not in the lesson but which I would put in any real build. Store the model name and the dimension count in the collection metadata when you create the store, and assert on them when you query. That is about ten lines of code, and it converts this entire category of silent failure into a loud one that fails on the first query. Given what we just watched, that is an extremely good trade.

The five things that go wrong, and what each one actually means.

OpenAI error, api key must be set. That is no dot env file, or the wrong variable name, or you forgot to call load dot env. The name has to be exactly OPENAI underscore API underscore KEY.

Rate limit error, quota exceeded. This one is badly named and sends people down the wrong path. It is not a rate limit. Your key works fine — the account simply has no credit. Add funds in Billing. If you are sitting there adding retry logic and backoff, you are solving a problem you do not have.

Module not found error. Your virtual environment is not active, or you installed into a different one. Check for the venv prefix in your prompt. This is the one that comes back to bite people who open a new terminal tab and forget to re-activate.

No dot txt files found. You are running the script from a different directory than the one holding docs. Relative paths are relative to where you launched python, not to where the file lives.

And created a chunk of size N longer than eight hundred. That is a warning, not an error. A single paragraph exceeded the target. Expected behaviour, as we saw. Nothing to fix.

Before you move on, check yourself against these. If you are watching a recording of this, pause here — genuinely try to answer them out loud before reading on.

One. Why does a two million token window not remove the need for RAG? Two reasons: scale, because real corpora are orders of magnitude larger; and cost and quality, because sending irrelevant context is expensive, slow, and produces worse answers.

Two. Four million tokens at a chunk size of five hundred. Eight thousand chunks, and eight thousand vectors. One vector per chunk, always.

Three. Ten chunks back, four irrelevant. Not broken. A retriever always returns k results whether or not they are any good.

Four. What is in the prompt? The user's question and the original English text of the top chunks. Never the vectors.

Five. Which Document attribute survives chunking? Metadata. Every chunk inherits its parent's.

Six. Chunk size counts characters, not tokens.

Seven. Leave out persist directory and Chroma runs in memory — everything vanishes when the script ends and you pay to embed again.

Eight. Documents with model A, queries with model B. The symptom is that there is no symptom. No error, collapsed scores, and confidently wrong chunks.

The whole session in ten lines. This is your revision card.

Models have a token limit, and real document stores are millions of times larger. So you retrieve the relevant pieces instead of sending everything.

A RAG system is two pipelines. Ingestion runs once. Retrieval runs per question.

Ingestion: cut documents into chunks, embed each chunk, store the vectors with their text.

An embedding is a fixed-length list of numbers where similar meaning gives similar numbers.

A vector database stores those numbers and searches them fast.

Retrieval: embed the question the same way, rank stored vectors by closeness, take the top k.

Send the model the question plus the original text of those chunks. Never the vectors.

Set chunk overlap so sentences on the seam survive, and set persist directory or you lose the work.

And the last one, which is the one that will actually cost you an afternoon: one embedding model, one dimension count, everywhere. Breaking this fails silently.

That is the ingestion pipeline, end to end. Five articles in; five hundred and forty-seven vectors on disk, searchable, each one still carrying the file it came from.

And one decision is now locked in: text-embedding-3-small at fifteen hundred and thirty-six dimensions. That applies to the whole project from here on.

Next is the retrieval pipeline properly — a question in, embedded with that same model, the closest chunks pulled out, a prompt built, and an LLM to answer it. Including what to do when the retriever comes back with nothing good.

If you want to run any of this, the whole repository is at ragverse dot d i y. The most useful thing you can do with it is point it at a set of documents you actually care about — everything here works unchanged.

Thanks for watching.