Two windows, eight demos

Slides on one side, a terminal on the other. Whenever you see an orange bar, I stop talking and run the command in it. The output on the slide is the real output from that command.

Several hundred internal documents

Policy guidelines. Technical specs. Customer support logs. Contracts.

Someone asks a question that one of those documents answers. The obvious move is to paste all of them into the model and ask.

That does not work — and the reason it does not work is the whole reason this lesson exists.

Retrieval Augmented Generation

The model does not get everything. It gets the right few pages.

That is the entire idea. Everything else in this session is mechanics.

Models do not read words

A token is the unit of text a model processes. Sometimes a whole word. Often a fragment of one. Models do not read letters and they do not read words.

One token is roughly three quarters of an English word. Common words are one token; rare or long words split into several. Spaces travel with the token that follows.

Read the scale carefully

Each step to the right is ten times larger, not one step larger. The gap between a model window and an enterprise archive is not a gap you close by waiting for bigger models.

You also pay per token

Even when everything does fit, sending it all is the wrong move.

Two pipelines

A chunk is a slice, not a section

Chunking is breaking large documents into small pieces. You choose the size. Set it to 1,000 tokens and 10 million tokens of documents becomes 10,000 chunks.

Why cut at all? Because nobody wants an entire Wikipedia article returned because one sentence in it matched. The chunk is your unit of retrieval — it is the smallest thing the system can hand back.

This is not an LLM

Different model, different job. It does not generate text. Text goes in; a list of numbers comes out.

Fixed length output. One word in or nine hundred words in, the vector that comes out is always the same length.

Does that actually hold?

Every group is tighter inside than out. That is a neighbourhood — and nobody labelled any of it.

The vector database stores both

Purpose built: Pinecone, Weaviate, Chroma, Qdrant. FAISS is a library rather than a service. Or add pgvector to a Postgres you already run.

The question takes the same road

1. The question goes through the same embedding model the documents went through.
2. It comes out as a vector of the same length.
3. The retriever compares that one vector against every stored vector.
4. It ranks them by closeness and returns the top k.

One secret, in one place

OPENAI_API_KEY=sk-proj-xxxxxxxxxxxxxxxxxxxx

Load the files

def load_documents(docs_path):
    # Fail loudly and early rather than silently loading nothing
    folder = Path(docs_path)
    if not folder.is_dir():
        raise FileNotFoundError(f"Directory not found: {docs_path}")

    documents = [
        Document(
            page_content=path.read_text(encoding="utf-8"),
            metadata={"source": f"{docs_path}/{path.name}"},
        )
        for path in sorted(folder.glob("*.txt"))   # sorted = reproducible
    ]

    if len(documents) == 0:
        raise ValueError(f"No .txt files found in {docs_path}")

    return documents

Chunk them

def split_documents(documents, chunk_size=800, chunk_overlap=0):
    splitter = CharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separator="\n\n",         # prefer to break at paragraph boundaries
    )
    return splitter.split_documents(documents)

What chunk_overlap actually does

With overlap at zero, a sentence that straddles the boundary ends up half in one chunk and half in another — and neither half matches the question well.

Rule of thumb: 10–20% of chunk size. For 800 characters, 100 is a reasonable default.

Try it: watch the chunks re-cut

Embed and store — one call does both

def create_vector_store(chunks, db_path):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

    vector_store = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=db_path,
        collection_metadata={"hnsw:space": "cosine"},
    )
    return vector_store

The three arguments that matter

Ask it something

All five from spacex.txt, out of a store containing four other companies. Your ingestion pipeline is finished and correct.

This is what reaches the model

Sent: the question, plus the original English text.  Not sent: the vectors. Their job finished the moment the matching was done.

The consistency rule

Same store. Same question. Nothing crashes.

Scores collapse from 0.66 to 0.06, and a Microsoft chunk is returned for a question about SpaceX.

Try it: flip the model, break the system

What this means in practice

1. Choose your embedding model before you ingest a single document.
2. Choose your dimension count at the same time, and write it down.
3. If you switch model later, you re-embed the entire corpus. There is no partial migration.
4. Changing dimensions within the same model breaks it too.

The whole session in ten lines

1. Models have a token limit. Real document stores are millions of times larger.
2. So you retrieve the relevant pieces instead of sending everything.
3. A RAG system is two pipelines. Ingestion runs once; retrieval runs per question.
4. Ingestion: cut documents into chunks, embed each chunk, store vectors with their text.
5. An embedding is a fixed-length list of numbers where similar meaning gives similar numbers.
6. A vector database stores those numbers and searches them fast.
7. Retrieval: embed the question the same way, rank by closeness, take the top k.
8. Send the model the question plus the original text. Never the vectors.
9. Set chunk_overlap so sentences on the seam survive. Set persist_directory or you lose the work.
10. One embedding model, one dimension count, everywhere. Breaking this fails silently.