Two windows, eight demos

Slides on one side, a terminal on the other. Whenever you see an orange bar, I stop talking and run the command in it. The output on the slide is the real output from that command.

Retrieval Augmented Generation

The model does not get everything. It gets the right few pages.

That is the entire idea. Everything else in this session is mechanics.

Where you have met one — and why

Models do not read words

A token is the unit of text a model processes. Sometimes a whole word. Often a fragment of one. Models do not read letters and they do not read words.

One token is roughly three quarters of an English word. Common words are one token; rare or long words split into several. Spaces travel with the token that follows.

Everything has to fit inside one window

The context window is the total number of tokens a model can hold in a single request. Everything you send and everything it writes back has to fit. Past that limit, the information is simply not there.

Read the scale carefully

Each step to the right is ten times larger, not one step larger. The gap between a model window and an enterprise archive is not a gap you close by waiting for bigger models.

Two pipelines

A chunk is a slice, not a section

Chunking is breaking large documents into small pieces. You choose the size. Set it to 1,000 tokens and 10 million tokens of documents becomes 10,000 chunks.

Why cut at all? Because nobody wants an entire Wikipedia article returned because one sentence in it matched. The chunk is your unit of retrieval — it is the smallest thing the system can hand back.

This is not an LLM

Different model, different job. It does not generate text. Text goes in; a list of numbers comes out.

Fixed length output. One word in or nine hundred words in, the vector that comes out is always the same length.

Similar meaning, similar numbers

A vector embedding is a list of numbers that stands in for a piece of text. Each number is called a dimension.

Real dimensions are not labelled and nobody knows what any single one means. These labels are a teaching device only.

Does that actually hold?

Every group is tighter inside than out. That is a neighbourhood — and nobody labelled any of it.

The vector database stores both

Purpose built: Pinecone, Weaviate, Chroma, Qdrant. FAISS is a library rather than a service. Or add pgvector to a Postgres you already run.

The question takes the same road

1. The question goes through the same embedding model the documents went through.
2. It comes out as a vector of the same length.
3. The retriever compares that one vector against every stored vector.
4. It ranks them by closeness and returns the top k.

Load the files

def load_documents(docs_path):
    # Fail loudly and early rather than silently loading nothing
    folder = Path(docs_path)
    if not folder.is_dir():
        raise FileNotFoundError(f"Directory not found: {docs_path}")

    documents = [
        Document(
            page_content=path.read_text(encoding="utf-8"),
            metadata={"source": f"{docs_path}/{path.name}"},
        )
        for path in sorted(folder.glob("*.txt"))   # sorted = reproducible
    ]

    if len(documents) == 0:
        raise ValueError(f"No .txt files found in {docs_path}")

    return documents

Five files in, five Documents out

Chunk them

def split_documents(documents, chunk_size=800, chunk_overlap=0):
    splitter = CharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separator="\n\n",         # prefer to break at paragraph boundaries
    )
    return splitter.split_documents(documents)

800 is a target, not a cap

What chunk_overlap actually does

With overlap at zero, a sentence that straddles the boundary ends up half in one chunk and half in another — and neither half matches the question well.

Rule of thumb: 10–20% of chunk size. For 800 characters, 100 is a reasonable default.

Embed and store — one call does both

def create_vector_store(chunks, db_path):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=db_path,
        collection_metadata={"hnsw:space": "cosine"},
    )
    return vector_store

One row per chunk

db_chroma/
├── chroma.sqlite3          text, metadata, ids
└── a1b2c3d4-.../           the collection
    ├── data_level0.bin     the vectors
    └── link_lists.bin      the search index

The original text is not optional. Without it you have numbers and nothing to send an LLM.

Ask it something

All five from spacex.txt, out of a store containing four other companies. Your ingestion pipeline is finished and correct.

This is what reaches the model

Sent: the question, plus the original English text.  Not sent: the vectors. Their job finished the moment the matching was done.

The consistency rule

Same store. Same question. Nothing crashes.

Scores collapse from 0.66 to 0.06, and a Microsoft chunk is returned for a question about SpaceX.

The whole session in ten lines

1. Models have a token limit. Real document stores are millions of times larger.
2. So you retrieve the relevant pieces instead of sending everything.
3. A RAG system is two pipelines. Ingestion runs once; retrieval runs per question.
4. Ingestion: cut documents into chunks, embed each chunk, store vectors with their text.
5. An embedding is a fixed-length list of numbers where similar meaning gives similar numbers.
6. A vector database stores those numbers and searches them fast.
7. Retrieval: embed the question the same way, rank by closeness, take the top k.
8. Send the model the question plus the original text. Never the vectors.
9. Set chunk_overlap so sentences on the seam survive. Set persist_directory or you lose the work.Overlap your chunks so sentences on the seam survive. Tell the database where to save, or you lose the work.
10. One embedding model, one dimension count, everywhere. Breaking this fails silently.