Retrieval Augmented
Generation

How to give a language model access to a hundred thousand documents without ever putting a hundred thousand documents in the prompt.

Wil A decade in AI · incentivised data training with NEAR AI · retrieval and ingestion systems in production

Follow along at ragverse.diy — these slides, five playgrounds you can drive yourself, and all the code

RETRIEVAL LESSON · the theory, then the code
YOU NEED · Python 3.10+, an editor, one OpenAI API key
COST · Well under a cent of embeddings
EIGHT DEMOS · one per concept, each one runnable

Quick word on who is talking. **I am Wil.** I have been working in AI for **about a decade** — which means I started well before GPTs made any of this fashionable, back when you built your own models and your own data pipelines. More recently I have been working on **incentivised data training structures with NEAR AI**. And running through all of it, I have built a lot of retrieval and ingestion systems, on a lot of different stacks. I mention that for one reason. **This is not a lesson I read and repeated.** It is a system I have built more than once, and **every mistake in it is one I have made.** Here is the problem in one line. **How do you give a language model access to a hundred thousand documents, without ever putting a hundred thousand documents in the prompt?** That is the problem. Everything after it is mechanics. So here is where we are going. **Why RAG exists**, and why waiting for a bigger model will not fix it. **The shape of the system** — two pipelines, one diagram. **What an embedding really is**, tested against a real model. Then **five real documents pushed all the way through**, so you see what each step produces. And at the end, **the mistake that breaks most first builds** — the one that raises no error at all. One thing before we start. Everything is at **ragverse dot d i y** — these slides, five playgrounds you can drive yourself, and all the code. Open it in another tab now if you want to follow along.

Part one

Why RAG exists

The problem is not that models are stupid.
The problem is that they are small.

00 · Definition

Retrieval Augmented Generation

A language model combined with a retrieval system. The retrieval system searches external sources — documents, databases, knowledge bases — and pulls only the relevant pieces into the prompt, at the moment the model needs them.

The model does not get everything. It gets the right few pages.

This is not a niche technique Almost every language-model product you have used is doing this underneath — a model that searches the web, an assistant that answers from your own files, a support bot that cites the article it used. Retrieval is the default way a model is given anything it was not trained on.

That is the entire idea. Everything else in this session is mechanics.

Picture this. You work somewhere with a few hundred internal documents, and somebody asks a question that **exactly one of them** answers. The obvious move is to paste them all in and ask. **That does not work** — and why it does not work is the whole lesson. So here is the definition. Retrieval Augmented Generation is **a language model plus a retrieval system**. The retrieval system searches your documents and pulls **only the relevant pieces** into the prompt, at the moment the model needs them. The model does not get everything. **It gets the right few pages.** And I want to plant something early here, because people hear "RAG" and file it away as **one technique among many.** It is not. **Almost every language-model product you have used is doing this underneath.** A model that searches the web and cites what it found. An assistant that answers from your own files. A support bot that links the article it used. **Retrieval is the default way a model is given anything it was not trained on.** That is the entire idea. Everything after this is mechanics.

00 · Why anyone builds one

Where you have met one — and why

Where you have met it

The support assistant that answers and links the help article.
A model that searches the web and cites what it found.
A coding assistant that answers about your repository.
Internal Q&A over policy, security questionnaires, runbooks.

Why teams build it

Your data. No model was trained on your contracts or your tickets.
Today's data. A policy changed this morning. Index it; done.
Receipts. The retriever knows which chunks it used, so the answer can point at them.

The question everyone asks: why not fine-tune? Fine-tuning teaches a model a style — a format, a voice, a way of reasoning. It is an expensive way to teach it a fact, and a hopeless way to teach it a fact that changes on Tuesday. Retrieval gives a model facts it can point at.

Let me make that concrete, and then answer the obvious question. You have met these. The support chatbot that answered you and then **linked the exact help article**. A coding assistant that answers about **your** repository rather than a generic one. Internal question-and-answer over a company's own policies. So why do teams build them? **Three reasons.** **The first is your data.** Nothing was trained on your contracts or your support tickets. Ask anyway, and a model will cheerfully invent something plausible. **The second is today's data.** A policy changed this morning. You are not going to retrain a model over that. **And the third is receipts.** The retriever knows exactly which chunks it used, so the answer can point at them. **An answer nobody can check is worth close to nothing.** Which answers the question I get every single time. **Why not just fine-tune?** Fine-tuning teaches a model **a style** — a format, a voice. It is an expensive way to teach it a fact, a hopeless way to teach it one that changes on Tuesday, and it gives you nothing to cite. **Fine-tuning changes how a model speaks. Retrieval changes what it knows.**

01 · Tokens

Models do not read words

A token is the unit of text a model processes. Sometimes a whole word. Often a fragment of one. Models do not read letters and they do not read words.

Counted with the cl100k_base tokenizer

text : 'Retrieval augmented generation is powerful' words : 5 tokens : 7 split as : Ret | rie | val | _augmented | _generation | _is | _powerful

One token is roughly three quarters of an English word. Common words are one token; rare or long words split into several. Spaces travel with the token that follows.

01 · The context window

Everything has to fit inside one window

The context window is the total number of tokens a model can hold in a single request. Everything you send and everything it writes back has to fit. Past that limit, the information is simply not there.

Where the frontier sits As of 2026 the major models have converged on 1 million tokens — GPT-5.5, Gemini 3.1 Pro, Claude Opus 4.8 and Sonnet 5 all sit there. A few advertise up to 10 million, though no published benchmark shows quality holding near that.
Verify before you quote it. These figures move every few months.

01 · The scale gap

Read the scale carefully

How much text, in tokens — logarithmic scale One chunk ███ 1,000 These 5 articles ████████ 72,169 A frontier model window ███████████ 1,000,000 The largest advertised ██████████████ 10,000,000 A mid sized company, 1 TB █████████████████████████ 250,000,000,000 An enterprise archive, 1 PB ██████████████████████████████████ 250,000,000,000,000

Each step to the right is ten times larger, not one step larger. The gap between a model window and an enterprise archive is not a gap you close by waiting for bigger models.

Read this one carefully, because it is **a logarithmic scale**. Each step to the right is **ten times** the last — not one step. Our five articles come to **seventy-two thousand tokens**. That fits inside a frontier model easily. And I will say the honest thing here: **for five documents you genuinely do not need RAG.** I would rather tell you that than pretend otherwise. Now watch what happens. A mid-sized company with a terabyte of documents: **two hundred and fifty billion tokens.** That is **two hundred and fifty thousand times** a model window. An enterprise archive is a thousand times larger again. **This is not a gap you close by waiting for bigger models.** And there is a second reason, which matters just as much. You pay per token — and burying the one relevant paragraph in half a million irrelevant ones gives you **worse** answers, not better.

Part two

The shape of the system

Two pipelines, not one.
Learn them separately and the rest is easy.

02 · The whole system in one picture

Two pipelines

Pipeline 01 · INGESTION · runs once, before any question

Sourcedocuments

→

Chunkingcut to size

→

Embeddingmodel

→

Vectorslists of numbers

→

Vectordatabase

Pipeline 02 · RETRIEVAL · runs on every question

Userquestion

→

Embeddingmodel

→

Retrieverranks by closeness

→

Top matchingchunks

→

LLM writesthe answer

Not a drawing shortcut The same embedding model appears in both rows. That is the single most important detail on this slide, and we will break it on purpose at the end to show you why.

**This is the slide to photograph.** A RAG system is **two pipelines, not one.** People collapse them into one in their heads, and **that is exactly where the mental model breaks.** **The top row is ingestion, and it runs once** — before anybody asks anything. Documents go in, they get chunked into small pieces, each piece goes through an embedding model and becomes a vector, and the vectors land in a database. **Then it stops.** **The bottom row is retrieval, and it runs on every single question.** The question becomes a vector. A retriever ranks every stored vector by closeness. The top matches come back. And those chunks, plus the question, go to the model. Now look at the two orange boxes. **The same embedding model appears in both rows.** That is not a drawing shortcut. **We are going to break it deliberately at the end** to show you why it matters.

03 · Chunking

A chunk is a slice, not a section

Chunking is breaking large documents into small pieces. You choose the size. Set it to 1,000 tokens and 10 million tokens of documents becomes 10,000 chunks.

The word "chunk" is doing work here Chunks are cut by size, not by meaning. They do not respect chapters, headings or arguments. A naive splitter will cut through the middle of a sentence or a table without hesitating.

Why cut at all? Because nobody wants an entire Wikipedia article returned because one sentence in it matched. The chunk is your unit of retrieval — it is the smallest thing the system can hand back.

03 · The embedding model

This is not an LLM

Different model, different job. It does not generate text. Text goes in; a list of numbers comes out.

Measured with text-embedding-3-small

one word 1 words in -> 1536 numbers out one sentence 8 words in -> 1536 numbers out a paragraph 337 words in -> 1536 numbers out [ -0.0616 +0.0554 +0.0004 +0.0124 +0.0016 -0.0123 ... ]

Fixed length output. One word in or nine hundred words in, the vector that comes out is always the same length.

Why it is always the same length

The model reads the text once and produces one vector per token, then pools them into a single vector. Pooling is what throws the length away.

Three properties worth knowing

Unit length. Every vector comes back normalised, so cosine similarity is just a dot product.
8,192 token cap. Longer input is truncated silently — another reason to chunk.
The dimensions are ordered. Keep the first 512 and renormalise and it still works — a third of the storage.

Next box: **the embedding model.** And here is the thing that trips everybody up. **This is not an LLM.** Different model, different job. Nothing is generated. There is no vocabulary to sample from. It is far smaller and far cheaper. **Text goes in, a list of numbers comes out.** That is the whole interface. Look at the three rows. One word in. Eight words in. **Three hundred and thirty-seven words in.** And every single time, **exactly fifteen hundred and thirty-six numbers** come back. So why is it always the same length? Because of what the model actually does. It reads the text and produces **one vector per token**, the way any transformer does. Then it **pools** them — averages them, roughly — into **a single vector**. **Pooling is the step that throws the length away.** **And that fixed length is the property the whole system rests on.** Any two pieces of text, however different in length, become two lists of the same size — and two lists of the same size can be compared with arithmetic. Three things about those numbers that are worth carrying. **They come back unit length.** I checked — the norm is one, to four decimal places. Which means **cosine similarity and a plain dot product give you the same answer.** Cosine is not a preference here. It is the cheapest thing that works. **There is a ceiling** — about **eight thousand tokens**. Send more than that and the tail is **silently truncated.** No error, no warning. Another reason to chunk. **And the dimensions are ordered by importance.** Keep the first **five hundred and twelve**, renormalise, and the rankings still hold — **a third of the storage.** That is also how we are going to fake a convincing failure later on.

04 · What an embedding is

Similar meaning, similar numbers

A vector embedding is a list of numbers that stands in for a piece of text. Each number is called a dimension.

Text with similar meaning produces similar numbers.
Nothing else about embeddings matters as much as that sentence.

Word	Dim 1 · size	Dim 2 · domesticated	Dim 3 · sound
cat	34	8	7.5
kitten	33	8	7.1
dog	40	8	6.9
elephant	210	62	1.2

Real dimensions are not labelled and nobody knows what any single one means. These labels are a teaching device only.

So what is an embedding? **A list of numbers standing in for a piece of text.** Each number is called **a dimension** — that is two more words for your list. And here is the sentence that matters more than any other today. **Text with similar meaning produces similar numbers.** That one sentence is why any of this works. Somebody types **"my card got declined."** The help article says **"payment failures."** **Not a single word in common.** A keyword search returns nothing at all. An embedding puts those two right next to each other, because they mean the same thing. **That is what you are buying.** Now, this table is a teaching device, and I want to be straight with you: **the numbers in it are invented.** Cat and kitten sit almost on top of each other. Dog shares the domestication value but differs on size. Elephant is far from all three. Real embeddings are not labelled, and nobody knows what any single dimension means. **So let us check whether that intuition survives contact with a real model.**

04 · Eight real words

Does that actually hold?

Real embeddings · cosine similarity · text-embedding-3-small

cat -> dog 0.603 ████████████████████████ cat -> kitten 0.570 ███████████████████████ cat -> coffee 0.385 ███████████████ cat -> tea 0.358 ██████████████ cat -> apple 0.357 ██████████████ cat -> elephant 0.319 █████████████ cat -> mango 0.264 ███████████ DOMESTIC ANIMALS inside 0.531 outside 0.316 DRINKS inside 0.611 outside 0.313

Every group is tighter inside than out. That is a neighbourhood — and nobody labelled any of it.

05 · Where the vectors live

The vector database stores both

ID	Vector	Original text	Source
0001	`[0.021, -0.884, ...]`	Refunds are issued within 30 days.	`policy.pdf`
0002	`[-0.412, 0.663, ...]`	Q1 revenue reached 4.2 million.	`sales.xlsx`
0003	`[0.907, 0.115, ...]`	Guest wifi resets every Monday.	`itwiki.md`

One row per chunk The vector is how it gets found. The original text is what gets used. Store only the numbers and you have nothing to send the model.

Purpose built: Pinecone, Weaviate, Chroma, Qdrant. FAISS is a library rather than a service. Or add pgvector to a Postgres you already run.

Part three

Retrieval, in principle

Ingestion is finished. Nothing above runs again
until your documents change.

06 · Retrieval, step by step

The question takes the same road

The question goes through the same embedding model the documents went through.
It comes out as a vector of the same length.
The retriever compares that one vector against every stored vector.
It ranks them by closeness and returns the top k.

— worked example, not a program run — THE QUESTION "What were our sales in the first quarter?" ↓ [-0.398, 0.671, 1.194, ...] same model, same dimensions 0.91 Q1 revenue reached 4.2 million. ┐ 0.88 Quarterly sales by region, Jan to Mar. ├ top 3 to the LLM 0.84 Revenue targets for the first quarter. ┘ 0.42 Annual headcount summary. 0.31 Refunds are issued within 30 days. 0.19 Guest wifi resets every Monday.

A question arrives, and **it takes the same road the documents took.** It goes through **the same embedding model** — and I am going to keep repeating that — and it becomes a vector of the same length. The retriever then scores **every stored vector** by closeness, and returns the top matches. How many come back is called **top k**. That is the last word worth collecting, and **you choose it.** But here is the half that people miss. **Ask for five and you get five — whether or not any of them are any good.** **A retriever has no concept of "nothing here fits."** It always returns k results, neatly ranked. And notice the winning chunk says **"revenue"** while the question says **"sales."** Different words entirely. A keyword search misses that. **Matching on meaning is what embeddings buy you.**

Part four

What actually happens to your documents

Five articles in. A searchable vector database out.
About sixty lines of Python and one API key.

Five articles go in. A searchable database comes out.
Three steps — and what each one actually produces.

07 · Step one: reading the documents

A document is its text, plus where it came from

Loaded 5 documents google.txt 68,012 characters microsoft.txt 67,411 characters nvidia.txt 62,402 characters spacex.txt 55,954 characters tesla.txt 93,916 characters .page_contentthe text 'Google LLC ( , GOO-gəl) is an American multinational...' .metadataits source {'source': 'docs/google.txt'}docs/google.txt

Learn these two attributes now .page_content is the entire text of the file as one long string. .metadata is a dictionary — here, the file we read it from. This object shows up everywhere from here on. Two halves, and you need both Every record is the text itself — the whole file as one long run of characters — and where it came from. The second half is what lets an answer cite its source.

**Step one. Read the documents in.** Five files go in, and **five records** come out. And each record has **two halves.** **The first half is the text itself** — the whole file, as one long run of characters. Sixty-eight thousand of them for Google. Ninety-four thousand for Tesla. **The second half is where it came from.** Right now that is just the filename — and **we put it there ourselves**, which means you can put anything you like alongside it. Page numbers. Authors. Who is allowed to see it. **Remember that second half**, because in a moment we are going to cut these five documents into hundreds of pieces — and **every single piece will carry it.** **That is how a system can tell you which file an answer came from**, when the thing it actually retrieved was a four-hundred-character fragment. So. Five documents. **Ninety thousand characters each, give or take.** Far too big to be useful. Let us cut them up.

08 · Step two: cutting them up

800 is a target, not a cap

Split 5 documents into 539 chunks chunks 539 smallest 15 characters largest 1,719 characters average 643 characters 79 of 539 chunks came out longer than the 800 target

Why CharacterTextSplitter splits on the separator first, then merges pieces up to chunk_size. It never breaks a paragraph in half to hit the number. So you will see warnings — they are warnings, not errors.
RecursiveCharacterTextSplitter handles this better and is what you would use in a real project. We stay on the simple one so the mechanics show. The splitter breaks on blank lines first, then glues the pieces back together up towards your target. What it will never do is cut a paragraph in half to hit the number — so a long paragraph gives you a long chunk.

**Step two. Cut them up.** I have asked for chunks of eight hundred characters, and **with overlap switched off for the moment** — I will come back to that in a second. Five documents in, **five hundred and thirty-nine chunks** out. Now look at the spread, because **this is where the mental model is usually wrong.** Smallest: **fifteen characters.** Largest: **seventeen hundred.** Average: **six hundred and forty-three.** We asked for eight hundred, and got almost nothing that is actually eight hundred. Here is why. The splitter **breaks on blank lines first**, and then glues the pieces back up towards your target. **What it will never do is cut a paragraph in half to hit the number.** So a long paragraph gives you a long chunk. **Chunk size is a target, not a cap.** Now — about that overlap setting I switched off. **It is the one that decides whether this whole system actually works.** Let me show you why.

08 · Overlap

What overlap actually does

chunk_overlap = 0overlap = 0 characters [0] Tesla was incorporated in July 2003 by Martin Eberhard and Marc Tarpenning as Tesla Motors. Its name is a tribute to the [1] inventor Nikola Tesla. In February 2004 Elon Musk led Tesla's... chunk_overlap = 40overlap = 40 characters [0] Tesla was incorporated in July 2003 by Martin Eberhard and Marc Tarpenning as Tesla Motors. Its name is a tribute to the [1] Motors. Its name is a tribute to the inventor Nikola Tesla. In...

With overlap at zero, a sentence that straddles the boundary ends up half in one chunk and half in another — and neither half matches the question well.

Rule of thumb: 10–20% of chunk size. For 800 characters, 100 is a reasonable default.

Here is the same splitter on a small piece of the Tesla article, blown up so you can read the seam. **Overlap zero — that is what produced the five hundred and thirty-nine chunks a moment ago.** Look at the seam. **A sentence is cut clean in half.** And here is the consequence. Neither half carries the whole fact, so neither one is a good match for a question about it. **The fact is sitting in your database, and it is unreachable.** **Now overlap forty.** Chunk one begins by repeating the tail of chunk zero — so **the sentence on the seam survives intact somewhere.** And that is all overlap does. Each chunk repeats the last n characters of the one before it. Rule of thumb: **ten to twenty percent of your chunk size.** These example chunks are small, so forty characters is enough to show it. **On the real run I use a hundred**, against a chunk size of eight hundred. **And that is what takes us from five hundred and thirty-nine chunks to five hundred and forty-seven.** Eight extra pieces, because the seams now repeat a little text. **That is the whole price** — a bit more storage, a few more embedding calls — and it is almost always worth paying.

09 · Step three: turning chunks into vectors

Every chunk becomes numbers, text and a source

Stored 547 vectors in db_chroma/ the collection now holds 547 vectors the vector 1,536 numbers [+0.0223, -0.0135, -0.0015, ...] the original text 'Google LLC ( , GOO-gəl) is an American multinational...' the metadata {'source': 'docs/google.txt'}

db_chroma/
├── chroma.sqlite3          text, metadata, ids
└── a1b2c3d4-.../           the collection
    ├── data_level0.bin     the vectors
    └── link_lists.bin      the search index

The original text is not optional. Without it you have numbers and nothing to send an LLM.

**Step three. Turn those five hundred and forty-seven chunks into numbers, and file them away.** And those two things happen together, in one pass — the embedding and the storing. **Five hundred and forty-seven vectors** on disk. Now look at what is kept for each one, because **this is that vector database table from earlier — now real.** **The vector:** fifteen hundred and thirty-six numbers. **The original text.** And **the source**, still saying docs slash google dot txt. **There it is** — the thing I asked you to remember two slides ago. It went in at the loader, came through the splitter untouched, and it is now sitting in the database attached to a chunk. And the line to leave with: **the original text is not optional.** Without it you have numbers, and nothing to send a model. **That is the ingestion pipeline finished.** Five articles went in; five hundred and forty-seven searchable vectors came out. **So let us ask it something.**

10 · Retrieval, for real

Ask it who founded SpaceX

Searched across 547 chunks of five real articles

question 'Who founded SpaceX?' collection 547 vectors 0.662 ███████████████ spacex.txt In early 2002, Elon Musk started to look... 0.628 ██████████████ spacex.txt SpaceX was founded in 2002 by Elon Musk... 0.627 ██████████████ spacex.txt Space Exploration Technologies Corp., ... 0.598 █████████████ spacex.txt The company is credited with advances... 0.591 █████████████ spacex.txt In early 2012, approximately two-thirds...

All five from spacex.txt, out of a store containing four other companies. Your ingestion pipeline is finished and correct.

11 · The actual prompt

This is what reaches the model

│ Answer the question using only the context below. │ │ CONTEXT: │ In early 2002, Elon Musk started to look for staff for his company, │ soon to be named SpaceX. Musk approached five people for the initial │ positions, including Griffin, who declined the position of Chief │ Engineer, Jim Cantrell and John Garvey ... │ │ SpaceX was founded in 2002 by Elon Musk with the goal of reducing │ spaceflight costs ... │ │ QUESTION: Who founded SpaceX? prompt length: 2,972 characters, built from the top 3 chunks

Sent: the question, plus the original English text. Not sent: the vectors. Their job finished the moment the matching was done.

And here is **the prompt.** Every character of it. An instruction. The word CONTEXT. **Three paragraphs of ordinary English** lifted straight out of the article. And the question. Just under three thousand characters. Now notice what is **not** in it. **There is nothing numeric here.** No vectors. No embeddings. No scores. **The model has no idea a vector database was ever involved.** And this is the step everybody misreads. **Vectors are only used for finding.** Their job ends the instant the matching is done — and then you are back in plain English. All of the machinery — the chunking, the embeddings, the fifteen hundred dimensions — **exists to produce those three paragraphs.** **The prompt at the end is boring. And it is meant to be.** So — **that is a complete, working system, end to end.** Documents in, a real answer out, with sources. Which makes this exactly the right moment to show you **the one way it falls apart without telling you.**

Part five

The mistake that breaks
most first builds

Nothing crashes. There is no error message.

12 · The consistency rule

The consistency rule

Use the same embedding model and the same dimension count for your documents and for your queries. Every time. No exceptions.

Documents embedded in January

text-embedding-3-large
3,072 dimensions

Queries embedded in March

text-embedding-3-small
1,536 dimensions

Think of them as separate languages A vector written by one model means nothing to another. The two systems cannot understand each other, and neither one will tell you.

Here is the rule. **Use the same embedding model, and the same dimension count, for your documents and for your queries.** Every time. No exceptions. I have been foreshadowing this all the way through. **The two orange boxes** on the diagram — the same model in both rows. And when the question took the same road as the documents, I said I would **keep repeating it.** This is why. And the way it goes wrong is **completely ordinary.** You embed your documents in January. Two months later you write the query side, and you reach for a model — maybe a cheaper one, maybe you just do not remember which one you used. Think of embedding models as **separate languages.** **A vector written by one means nothing at all to another.** And here is the sting. The numbers are the same shape. The same range. They look entirely normal. **And neither system will tell you.**

12 · Break it on purpose

Same store. Same question. Nothing crashes.

The same 547 chunks · only the query model changed

question: 'Who founded SpaceX?' (answer lives in spacex.txt) RIGHT model text-embedding-3-small 0.662 spacex.txt In early 2002, Elon Musk started to look... 0.628 spacex.txt SpaceX was founded in 2002 by Elon Musk... 0.627 spacex.txt Space Exploration Technologies Corp., ... WRONG model text-embedding-3-large@1536 0.064 spacex.txt === Starbase, Texas === SpaceX manufact... 0.055 spacex.txt In December 2022, the U.S. Federal Comm... 0.052 microsoft.txt Microsoft became the third publicly ... ← WRONG FILE

Scores collapse from 0.66 to 0.06, and a Microsoft chunk is returned for a question about SpaceX.

So let us **break it on purpose.** And let us make it a fair test. The wrong model is asked for **the same fifteen hundred and thirty-six dimensions**, so the vector is **exactly the same shape.** The database has nothing to complain about. **Top block, the right model:** all three results from spacex dot txt. **Bottom block** — same database, same question. **Only the model that embedded the question changed.** **The scores collapsed by a factor of ten.** And the third result is from **microsoft dot txt**, returned for a question about SpaceX. Now notice **what did not happen.** **Nothing crashed.** No exception. No warning. No log line. Three results, ranked, formatted identically to the correct run. **Every piece of monitoring you have says the system is fine** — and a model handed those chunks writes a confident, wrong answer. And that is why this breaks first builds. Not because it is hard to fix — **because it is invisible until somebody checks an answer by hand.**

13 · Summary

The whole session in ten lines

Models have a token limit. Real document stores are millions of times larger.
So you retrieve the relevant pieces instead of sending everything.
A RAG system is two pipelines. Ingestion runs once; retrieval runs per question.
Ingestion: cut documents into chunks, embed each chunk, store vectors with their text.
An embedding is a fixed-length list of numbers where similar meaning gives similar numbers.
A vector database stores those numbers and searches them fast.
Retrieval: embed the question the same way, rank by closeness, take the top k.
Send the model the question plus the original text. Never the vectors.
Set chunk_overlap so sentences on the seam survive. Set persist_directory or you lose the work.Overlap your chunks so sentences on the seam survive. Tell the database where to save, or you lose the work.
One embedding model, one dimension count, everywhere. Breaking this fails silently.

**The whole lesson, on one slide.** This is your revision card. Models have a token limit, and real document stores are **millions of times larger.** So you **retrieve the relevant pieces** instead of sending everything. A RAG system is **two pipelines** — ingestion runs once, retrieval runs on every question. **Ingestion:** chunk, embed, and store the vectors **with their text.** An embedding is a **fixed-length list of numbers**, where **similar meaning gives similar numbers.** **Retrieval:** embed the question **the same way**, rank by closeness, and take the top k. Send the model the question plus **the original text** — never the vectors. Overlap your chunks so sentences on the seam survive. And tell the database where to save, or the work is gone when the process ends. And the one that will actually cost you an afternoon: **one embedding model, one dimension count, everywhere. Breaking that fails silently.**

That is Retrieval Augmented Generation

You take a question, embed it with that same model, pull the closest chunks out of db_chroma/, and hand them to an LLM to answer.

Two pipelines. Chunk, embed, store the text with the numbers. Ask with the same model, rank by closeness, send the paragraphs — never the vectors.

THE RESULT · 547 vectors on disk, searchable, with sources
LOCKED IN · text-embedding-3-small at 1,536 dimensions
BRING · a set of documents you actually want to ask questions about
YOU CAN NOW · draw both pipelines and say what each box is for
YOU KNOW THE TRAP · one embedding model, one dimension count, everywhere
TO GO FURTHER · the playgrounds, the code, and the long version
ALL OF IT · ragverse.diy

**So — that is Retrieval Augmented Generation.** Not a niche technique. **The default way a language model is given anything it was not trained on** — and now you know what is happening underneath every product that does it. You can draw both pipelines from memory. You can say what every box is for. You can predict roughly what a retriever will hand back, and **you know the one failure that will not announce itself.** That is genuinely most of it. If you want to go further, it is all at **ragverse dot d i y** — five playgrounds you can take apart in the browser, the code if you want to build it yourself, and a longer version if you want every aside. And the most useful thing you can do with it is **point it at documents you actually care about.** Everything you have seen today works unchanged. **Thanks for watching.**

Retrieval AugmentedGeneration

Why RAG exists

Retrieval Augmented Generation

Where you have met one — and why

Models do not read words

Everything has to fit inside one window

Read the scale carefully

The shape of the system

Two pipelines

A chunk is a slice, not a section

This is not an LLM

Similar meaning, similar numbers

Does that actually hold?

The vector database stores both

Retrieval, in principle

The question takes the same road

What actually happens to your documents

A document is its text, plus where it came from

800 is a target, not a cap

What overlap actually does

Every chunk becomes numbers, text and a source

Ask it who founded SpaceX

This is what reaches the model

The mistake that breaksmost first builds

The consistency rule

Documents embedded in January

Queries embedded in March

Same store. Same question. Nothing crashes.

The whole session in ten lines

That is Retrieval Augmented Generation

Retrieval Augmented
Generation

The mistake that breaks
most first builds