RAG, the fast way.

Quick word on who is talking. I am Wil.

I have been working in AI for about a decade — which means I started well before GPTs made any of this fashionable, back when you built your own models and your own data pipelines.

More recently I have been working on incentivised data training structures with NEAR AI. And running through all of it, I have built a lot of retrieval and ingestion systems, on a lot of different stacks.

I mention that for one reason. This is not a lesson I read and repeated. It is a system I have built more than once, and every mistake in it is one I have made.

Here is the problem in one line. How do you give a language model access to a hundred thousand documents, without ever putting a hundred thousand documents in the prompt?

That is the problem. Everything after it is mechanics.

So here is where we are going. Why RAG exists, and why waiting for a bigger model will not fix it. The shape of the system — two pipelines, one diagram. What an embedding really is, tested against a real model.

Then five real documents pushed all the way through, so you see what each step produces. And at the end, the mistake that breaks most first builds — the one that raises no error at all.

One thing before we start. Everything is at ragverse dot d i y — these slides, five playgrounds you can drive yourself, and all the code. Open it in another tab now if you want to follow along.

One word on how this runs. Wherever a terminal block appears on a slide, the line above it names the command that printed it.

Behind them is a repository: eight numbered scripts, one per concept, and a finished sixty-line pipeline, all behind a single entry point. Whether the demos get run here or you run them afterwards, the output is the same. The address is on the last slide.

The scripts stand alone, except that seven and eight need the database the sixth one builds.

Picture a company with a few hundred internal documents. Policies, specs, support logs, contracts. Ordinary things, none of them written for a machine to read.

Nobody has read all of them. Whoever wrote any one of them has probably left. And the answer to a question somebody asks this afternoon is sitting in exactly one of them, and no one knows which.

That is the situation this is built for. Hold onto it — the next slide is the whole idea in a sentence.

Picture this. You work somewhere with a few hundred internal documents, and somebody asks a question that exactly one of them answers.

The obvious move is to paste them all in and ask. That does not work — and why it does not work is the whole lesson.

So here is the definition. Retrieval Augmented Generation is a language model plus a retrieval system.

The retrieval system searches your documents and pulls only the relevant pieces into the prompt, at the moment the model needs them.

The model does not get everything. It gets the right few pages.

And I want to plant something early here, because people hear "RAG" and file it away as one technique among many. It is not.

Almost every language-model product you have used is doing this underneath. A model that searches the web and cites what it found. An assistant that answers from your own files. A support bot that links the article it used.

Retrieval is the default way a model is given anything it was not trained on.

That is the entire idea. Everything after this is mechanics.

Let me make that concrete, and then answer the obvious question.

You have met these. The support chatbot that answered you and then linked the exact help article. A coding assistant that answers about your repository rather than a generic one. Internal question-and-answer over a company's own policies.

So why do teams build them? Three reasons.

The first is your data. Nothing was trained on your contracts or your support tickets. Ask anyway, and a model will cheerfully invent something plausible.

The second is today's data. A policy changed this morning. You are not going to retrain a model over that.

And the third is receipts. The retriever knows exactly which chunks it used, so the answer can point at them. An answer nobody can check is worth close to nothing.

Which answers the question I get every single time. Why not just fine-tune?

Fine-tuning teaches a model a style — a format, a voice. It is an expensive way to teach it a fact, a hopeless way to teach it one that changes on Tuesday, and it gives you nothing to cite.

Fine-tuning changes how a model speaks. Retrieval changes what it knows.

The first word you need is token.

A token is the unit of text a model actually processes. Sometimes that is a whole word. Often it is a fragment of one.

Models do not read letters. They do not read words. They read tokens.

And that matters for a very practical reason. Every limit you will ever hit, and every price you will ever pay, is counted in tokens.

Look at the output. Five words went in. Seven tokens came back.

"Retrieval" split into three pieces. "Augmented" and "powerful" survived whole. Common words are one token; rarer ones fragment.

So here is the rule of thumb to carry: one token is about three quarters of an English word.

Second word: the context window.

That is the total number of tokens a model can hold in one request. Everything you send, and everything it writes back, sharing one budget.

Past that limit, the information is simply not there.

In 2026 the major models have converged on one million tokens. A few advertise ten million — though no benchmark shows quality holding near the top of those windows.

A million sounds like plenty. So let us see how it compares to what a company actually holds.

Read this one carefully, because it is a logarithmic scale. Each step to the right is ten times the last — not one step.

Our five articles come to seventy-two thousand tokens. That fits inside a frontier model easily.

And I will say the honest thing here: for five documents you genuinely do not need RAG. I would rather tell you that than pretend otherwise.

Now watch what happens. A mid-sized company with a terabyte of documents: two hundred and fifty billion tokens. That is two hundred and fifty thousand times a model window. An enterprise archive is a thousand times larger again.

This is not a gap you close by waiting for bigger models.

And there is a second reason, which matters just as much. You pay per token — and burying the one relevant paragraph in half a million irrelevant ones gives you worse answers, not better.

There is a second reason, and people forget it because the first one is so dramatic.

Even when everything does fit, sending all of it is still wrong. You pay per token, so you pay for every irrelevant word. Latency scales with what you send. And — this is the counterintuitive one — you get worse answers.

Bury the one relevant paragraph inside five hundred thousand tokens and the model has a harder job than if you had handed it the paragraph. More context is not more helpful. Relevant context is.

This is the slide to photograph.

A RAG system is two pipelines, not one. People collapse them into one in their heads, and that is exactly where the mental model breaks.

The top row is ingestion, and it runs once — before anybody asks anything. Documents go in, they get chunked into small pieces, each piece goes through an embedding model and becomes a vector, and the vectors land in a database. Then it stops.

The bottom row is retrieval, and it runs on every single question. The question becomes a vector. A retriever ranks every stored vector by closeness. The top matches come back. And those chunks, plus the question, go to the model.

Now look at the two orange boxes. The same embedding model appears in both rows.

That is not a drawing shortcut. We are going to break it deliberately at the end to show you why it matters.

Next word: the chunk.

Chunking is cutting documents into small pieces, and you choose the size.

Here is the nuance that matters. A chunk is a slice, not a section. Chunks are cut by size, not by meaning. They do not respect headings or arguments — a naive splitter will cut through the middle of a sentence without hesitating.

So why cut at all? Because the chunk is your unit of retrieval — the smallest thing the system can hand back.

Make a whole document one chunk, and a question about one line drags the entire forty-page employee handbook into the prompt, because one sentence in it matched.

And now you are back to stuffing the context window — which is the thing you were trying to avoid.

Next box: the embedding model. And here is the thing that trips everybody up.

This is not an LLM. Different model, different job. Nothing is generated. There is no vocabulary to sample from. It is far smaller and far cheaper. Text goes in, a list of numbers comes out. That is the whole interface.

Look at the three rows. One word in. Eight words in. Three hundred and thirty-seven words in. And every single time, exactly fifteen hundred and thirty-six numbers come back.

So why is it always the same length? Because of what the model actually does. It reads the text and produces one vector per token, the way any transformer does. Then it pools them — averages them, roughly — into a single vector. Pooling is the step that throws the length away.

And that fixed length is the property the whole system rests on. Any two pieces of text, however different in length, become two lists of the same size — and two lists of the same size can be compared with arithmetic.

Three things about those numbers that are worth carrying.

They come back unit length. I checked — the norm is one, to four decimal places. Which means cosine similarity and a plain dot product give you the same answer. Cosine is not a preference here. It is the cheapest thing that works.

There is a ceiling — about eight thousand tokens. Send more than that and the tail is silently truncated. No error, no warning. Another reason to chunk.

And the dimensions are ordered by importance. Keep the first five hundred and twelve, renormalise, and the rankings still hold — a third of the storage. That is also how we are going to fake a convincing failure later on.

So what is an embedding? A list of numbers standing in for a piece of text. Each number is called a dimension — that is two more words for your list.

And here is the sentence that matters more than any other today.

Text with similar meaning produces similar numbers.

That one sentence is why any of this works. Somebody types "my card got declined." The help article says "payment failures."

Not a single word in common. A keyword search returns nothing at all. An embedding puts those two right next to each other, because they mean the same thing.

That is what you are buying.

Now, this table is a teaching device, and I want to be straight with you: the numbers in it are invented. Cat and kitten sit almost on top of each other. Dog shares the domestication value but differs on size. Elephant is far from all three.

Real embeddings are not labelled, and nobody knows what any single dimension means. So let us check whether that intuition survives contact with a real model.

Eight ordinary words, embedded with a real model, every pair measured. One point zero would mean identical.

Cat to dog: zero point six. Cat to kitten: zero point five seven. Then a clear drop to the fruit and the drinks.

Now — honestly, that contradicts the tidy table. The table said kitten should be closest.

The real model puts dog first, because cat and dog co-occur in text constantly, while "kitten" is a much narrower word. So do not over-read the ordering inside a cluster. And remember that real systems embed paragraphs, not single words.

But look at the bottom two lines, because that is where the claim holds. Every group is tighter inside than outside.

That is a neighbourhood — and nobody labelled any of it.

Last box in the top row: the vector database. It stores embeddings, and it finds the closest ones fast.

But the detail people miss is right there in the table. It stores the original text alongside every vector.

The vector is how a chunk gets found. The original text is what actually gets used.

Store only the numbers, and you have a beautiful index and nothing to send a model — because you cannot turn a vector back into English.

That fourth column, the source, is metadata. And that is what lets you tell a user which file an answer came from.

If you are wondering about options: Pinecone, Weaviate, Chroma, Qdrant. We use Chroma here because it runs locally with no account. And if you already run Postgres, pgvector is often the right answer.

A question arrives, and it takes the same road the documents took.

It goes through the same embedding model — and I am going to keep repeating that — and it becomes a vector of the same length.

The retriever then scores every stored vector by closeness, and returns the top matches.

How many come back is called top k. That is the last word worth collecting, and you choose it.

But here is the half that people miss. Ask for five and you get five — whether or not any of them are any good.

A retriever has no concept of "nothing here fits." It always returns k results, neatly ranked.

And notice the winning chunk says "revenue" while the question says "sales." Different words entirely. A keyword search misses that. Matching on meaning is what embeddings buy you.

One more beat on top k, because this is the thing that catches people building their first system.

They ask something the documents genuinely do not answer, get five confident-looking chunks back, and conclude the retriever is broken. It is not. Ten chunks of which four are irrelevant is the expected shape of the output, not a bug. They are the least-bad of a bad set, and they look exactly like good ones.

What you do about it — filter on a score threshold, or re-rank — is a later topic. Knowing that it happens is not.

The API key. Create one on the OpenAI platform, copy it immediately — you cannot view it again — and put it in a file called dot env.

Two things that catch people out, and both are worth doing before you write a line.

Add credit. The API is prepaid and entirely separate from a ChatGPT subscription; paying for Plus gives you no API credit, which catches people constantly. Embedding these five documents costs well under a cent.

And add dot env to your gitignore before your first commit. A leaked key gets scraped off a public repository within minutes, and removing it from the working tree does not remove it from your history.

Step one: load the files. And here I depart from most tutorials deliberately.

Every RAG tutorial uses DirectoryLoader from langchain-community. That package was archived in May twenty twenty-six. And look at what it actually did for text files: glob a pattern, read each file, wrap the text in a Document with the path as metadata. That is the whole job — so we write it directly, in six lines.

Same objects out, one fewer dead dependency, and nothing hidden behind the word "loader".

Note the two guard clauses. One checks the folder exists; the other checks we actually loaded something. Without the second, an empty folder gives you a pipeline that runs happily, embeds nothing, and reports success.

Step one. Read the documents in.

Five files go in, and five records come out. And each record has two halves.

The first half is the text itself — the whole file, as one long run of characters. Sixty-eight thousand of them for Google. Ninety-four thousand for Tesla.

The second half is where it came from. Right now that is just the filename — and we put it there ourselves, which means you can put anything you like alongside it. Page numbers. Authors. Who is allowed to see it.

Remember that second half, because in a moment we are going to cut these five documents into hundreds of pieces — and every single piece will carry it.

That is how a system can tell you which file an answer came from, when the thing it actually retrieved was a four-hundred-character fragment.

So. Five documents. Ninety thousand characters each, give or take. Far too big to be useful. Let us cut them up.

Step two: chunk them. Articles of sixty thousand characters are useless units of retrieval — nobody wants a whole article back because one sentence matched.

Three arguments: chunk size, chunk overlap, and a separator set to a blank line so we prefer paragraph boundaries.

And the thing to flag hardest: chunk size of eight hundred means eight hundred characters, not tokens. Roughly two hundred tokens. Different splitters count in different units, and getting that wrong by a factor of four is easy.

You will also see the chunks are not all eight hundred. The splitter cuts on the separator first, then merges up to the target — it never breaks a paragraph to hit the number. So it is a target, not a cap.

Step two. Cut them up.

I have asked for chunks of eight hundred characters, and with overlap switched off for the moment — I will come back to that in a second.

Five documents in, five hundred and thirty-nine chunks out.

Now look at the spread, because this is where the mental model is usually wrong.

Smallest: fifteen characters. Largest: seventeen hundred. Average: six hundred and forty-three. We asked for eight hundred, and got almost nothing that is actually eight hundred.

Here is why. The splitter breaks on blank lines first, and then glues the pieces back up towards your target.

What it will never do is cut a paragraph in half to hit the number. So a long paragraph gives you a long chunk.

Chunk size is a target, not a cap.

Now — about that overlap setting I switched off. It is the one that decides whether this whole system actually works. Let me show you why.

Here is the same splitter on a small piece of the Tesla article, blown up so you can read the seam.

Overlap zero — that is what produced the five hundred and thirty-nine chunks a moment ago. Look at the seam. A sentence is cut clean in half.

And here is the consequence. Neither half carries the whole fact, so neither one is a good match for a question about it. The fact is sitting in your database, and it is unreachable.

Now overlap forty. Chunk one begins by repeating the tail of chunk zero — so the sentence on the seam survives intact somewhere.

And that is all overlap does. Each chunk repeats the last n characters of the one before it.

Rule of thumb: ten to twenty percent of your chunk size. These example chunks are small, so forty characters is enough to show it. On the real run I use a hundred, against a chunk size of eight hundred.

And that is what takes us from five hundred and thirty-nine chunks to five hundred and forty-seven. Eight extra pieces, because the seams now repeat a little text. That is the whole price — a bit more storage, a few more embedding calls — and it is almost always worth paying.

Same two numbers you just saw in code, except now you can move them.

Drag chunk size down and the count climbs while the average drops. These are precise, but read them — they have lost their surroundings. A chunk saying "in 2008 he was named chief executive" does not say who "he" is.

Drag it up and each chunk carries plenty of context, but a question about one detail now matches a chunk that is mostly about something else, and you send three times the tokens.

Then drag the overlap up and watch the highlight: that repeated text is what you are storing and paying to embed twice.

There is no correct answer here — only a shape. Big enough to stand alone, small enough to be about one thing.

Step three: embed and store. Two conceptual boxes, one line of code — Chroma dot from underscore documents embeds every chunk and writes it to disk in a single call.

This is also the only step that costs money and takes time. Every chunk is an API call. Five hundred and forty-seven chunks, thirty seconds to a couple of minutes, a fraction of a cent.

Three arguments matter. The embedding model — write that choice down, because the next lesson must use exactly the same one. Persist directory, or Chroma runs in memory and everything vanishes when the script ends. And cosine as the distance measure, which is the standard for text.

Look at what is stored per chunk: the vector, the original text, and the metadata — the source surviving all the way from the loader.

Step three. Turn those five hundred and forty-seven chunks into numbers, and file them away.

And those two things happen together, in one pass — the embedding and the storing.

Five hundred and forty-seven vectors on disk.

Now look at what is kept for each one, because this is that vector database table from earlier — now real.

The vector: fifteen hundred and thirty-six numbers. The original text. And the source, still saying docs slash google dot txt.

There it is — the thing I asked you to remember two slides ago. It went in at the loader, came through the splitter untouched, and it is now sitting in the database attached to a chunk.

And the line to leave with: the original text is not optional. Without it you have numbers, and nothing to send a model.

That is the ingestion pipeline finished. Five articles went in; five hundred and forty-seven searchable vectors came out. So let us ask it something.

Three arguments matter here, and each has a failure attached.

The embedding model. Write that choice down — the query side must use exactly the same one, and that is the finale of this lesson.

Persist directory. Leave it out and Chroma runs in memory: everything vanishes when the script ends, and you pay to embed again. The code looks like it worked, printed success, and there is simply nothing on disk.

And the distance measure, cosine, which is the standard for text because it compares the angle rather than the magnitude — so a long document and a short one about the same topic still score as similar.

The store has five hundred and forty-seven vectors covering five companies. And the question is "who founded SpaceX."

Every single result comes from spacex dot txt.

Not one Tesla chunk — despite Elon Musk appearing all over the Tesla article. That is exactly the near-miss that would trip a keyword search.

One note on the scores, because I know what you are thinking. Zero point six six is the top match, and that might feel low. It is not.

Cosine similarity between a short question and a long paragraph rarely exceeds zero point seven.

What matters is the gap between the top results and the rest — not the absolute number.

But the retrieved chunks are not the answer. They are what gets sent. So let us look at exactly what reaches the model.

And here is the prompt. Every character of it.

An instruction. The word CONTEXT. Three paragraphs of ordinary English lifted straight out of the article. And the question. Just under three thousand characters.

Now notice what is not in it. There is nothing numeric here. No vectors. No embeddings. No scores. The model has no idea a vector database was ever involved.

And this is the step everybody misreads. Vectors are only used for finding. Their job ends the instant the matching is done — and then you are back in plain English.

All of the machinery — the chunking, the embeddings, the fifteen hundred dimensions — exists to produce those three paragraphs.

The prompt at the end is boring. And it is meant to be.

So — that is a complete, working system, end to end. Documents in, a real answer out, with sources. Which makes this exactly the right moment to show you the one way it falls apart without telling you.

Here is the rule. Use the same embedding model, and the same dimension count, for your documents and for your queries. Every time. No exceptions.

I have been foreshadowing this all the way through. The two orange boxes on the diagram — the same model in both rows. And when the question took the same road as the documents, I said I would keep repeating it. This is why.

And the way it goes wrong is completely ordinary. You embed your documents in January. Two months later you write the query side, and you reach for a model — maybe a cheaper one, maybe you just do not remember which one you used.

Think of embedding models as separate languages. A vector written by one means nothing at all to another.

And here is the sting. The numbers are the same shape. The same range. They look entirely normal. And neither system will tell you.

So let us break it on purpose.

And let us make it a fair test. The wrong model is asked for the same fifteen hundred and thirty-six dimensions, so the vector is exactly the same shape. The database has nothing to complain about.

Top block, the right model: all three results from spacex dot txt.

Bottom block — same database, same question. Only the model that embedded the question changed.

The scores collapsed by a factor of ten. And the third result is from microsoft dot txt, returned for a question about SpaceX.

Now notice what did not happen.

Nothing crashed. No exception. No warning. No log line. Three results, ranked, formatted identically to the correct run.

Every piece of monitoring you have says the system is fine — and a model handed those chunks writes a confident, wrong answer.

And that is why this breaks first builds. Not because it is hard to fix — because it is invisible until somebody checks an answer by hand.

And here is the failure as a switch you can flip.

The documents were embedded once, correctly. The only thing this toggle changes is which model embeds the question.

Same model: top score around zero point five five, and the right answer clearly first.

Different model: same store, same question, same six facts — and the top score collapses to zero point zero two six. On six facts it happens to keep the right one near the top, which is exactly why this is so dangerous on a small test set. On five hundred chunks it returns a Microsoft paragraph for a question about SpaceX.

Now look at the errors-raised number. Zero. That is what a silent failure looks like.

So what does that mean in practice? Four rules.

Choose your embedding model before you ingest a single document. This is architecture, not an implementation detail you defer.

Choose the dimension count at the same time and write it down somewhere your future self will look.

If you switch model later, you re-embed everything. There is no partial migration — you cannot have half your corpus in each, because they cannot be compared.

And this one catches the careful: changing dimensions within the same model breaks it too. Same model, different language.

One suggestion not in the lesson: store the model name and dimension count in the collection metadata and assert on it when you query. Ten lines that turn a silent failure into a loud one.

The whole lesson, on one slide. This is your revision card.

Models have a token limit, and real document stores are millions of times larger. So you retrieve the relevant pieces instead of sending everything.

A RAG system is two pipelines — ingestion runs once, retrieval runs on every question.

Ingestion: chunk, embed, and store the vectors with their text.

An embedding is a fixed-length list of numbers, where similar meaning gives similar numbers.

Retrieval: embed the question the same way, rank by closeness, and take the top k.

Send the model the question plus the original text — never the vectors.

Overlap your chunks so sentences on the seam survive. And tell the database where to save, or the work is gone when the process ends.

And the one that will actually cost you an afternoon: one embedding model, one dimension count, everywhere. Breaking that fails silently.

So — that is Retrieval Augmented Generation.

Not a niche technique. The default way a language model is given anything it was not trained on — and now you know what is happening underneath every product that does it.

You can draw both pipelines from memory. You can say what every box is for. You can predict roughly what a retriever will hand back, and you know the one failure that will not announce itself. That is genuinely most of it.

If you want to go further, it is all at ragverse dot d i y — five playgrounds you can take apart in the browser, the code if you want to build it yourself, and a longer version if you want every aside.

And the most useful thing you can do with it is point it at documents you actually care about. Everything you have seen today works unchanged.

Thanks for watching.