← Claude Operator Track

Large knowledge — RAG vs stuffing the prompt

> When docs don’t fit, retrieve the right chunks — don’t drown the model in PDFs.

Canonical: https://www.brainyxai.co.za/education/claude/rag-vs-context

Markdown: https://www.brainyxai.co.za/md/education/claude/rag-vs-context.md

Course: Claude Operator Track

Lesson: 8 of 9

Minutes: 15

Author: Brainyx AI

What you will be able to do

• Know when prompt-stuffing is enough
• Explain RAG as chunk → embed → retrieve → generate
• List failure modes (wrong chunk, stale index, citation theater)

The problem

Business knowledge is bigger than a context window — and even when it fits, noise kills accuracy. You have three broad strategies:

| Strategy | Use when | Watch-outs |

| --- | --- | --- |

| Stuff selected docs into the prompt | Small, stable set | Manual curation tax |

| RAG (retrieve then generate) | Many docs, frequent questions | Chunking & stale indexes |

| Fine-tuning / distillation | Style or narrow task at scale | Cost; rarely first move for KB Q&A |

RAG, operator-level

1. Chunk documents (by structure when you can: headings, sections)

2. Embed chunks into vectors

3. Retrieve top-k for a question

4. Generate an answer grounded in those chunks

5. Cite what was used; fail closed when retrieval is weak

Failure modes we see in the wild

• Beautiful answers citing the wrong policy PDF
• Indexes that weren’t rebuilt after the handbook changed
• Chunks that split tables so numbers lie
• “Chat with your PDF” demos that never get evaluation harnesses

Brainyx AI angle

We implement RAG and knowledge agents as owned systems (your DB, your access control, your eval set). Use Claude to prototype prompts and chunking strategies; don’t confuse a demo notebook with go-live.

Official Academy modules on RAG / embeddings exist on [Anthropic Academy](https://anthropic.skilljar.com/) — take coding labs there; keep proprietary docs in your own infra.

Knowledge strategy lab

1. Pick one knowledge corpus (policies, product FAQ, engineering runbooks).

2. Decide: stuff, RAG, or hybrid — write three bullets justifying it.

3. Write five evaluation questions with known answers; you’ll reuse them when you build.

Checkpoints

• I have an eval set before I trust RAG
• I know stuffing ≠ a knowledge platform

When stuffing the prompt is the right answer

RAG is architecture, and architecture has running costs. Before building a pipeline, check whether you need one. If your knowledge fits comfortably in context, changes rarely, and is used by a handful of people, paste it in. A single well-constructed prompt with the source document attached beats a badly-tuned retrieval system on both accuracy and maintenance.

Build retrieval when at least one of these is true: the corpus is too large to fit, it changes often enough that re-pasting is a chore, you need citations back to source documents, or different users must see different subsets.

Retrieval quality is a chunking problem first

Most disappointing RAG systems fail at retrieval, not generation. The model produced a reasonable answer from the wrong three paragraphs.

Chunking decides what can be retrieved. Chunks that split mid-clause lose meaning; chunks that swallow a whole document retrieve everything and discriminate nothing. Respect document structure — sections, clauses, headings — rather than cutting at a fixed character count, and keep enough surrounding context that a retrieved chunk is interpretable on its own.

When answers are wrong, inspect what was retrieved before you touch the prompt. If the right chunk was never fetched, no amount of prompt engineering will rescue the answer.

Evaluating instead of hoping

Write thirty real questions with known correct answers before launch. For each, record whether the correct source was retrieved and whether the answer was right. That gives you two separate numbers — retrieval accuracy and generation accuracy — and they fail for different reasons.

Rerun this set after every meaningful change to chunking, embeddings, or prompts. Without it, you are making changes on vibes and discovering regressions through customer complaints.

Citation theatre

A system that displays sources looks trustworthy. A system that displays sources it did not actually use is worse than one showing none, because it manufactures confidence. Verify that displayed citations correspond to the retrieved chunks that informed the answer, and spot-check regularly.

South African context

Two local realities shape SA deployments. First, multilingual content: policy documents in English with customer queries in isiZulu, Afrikaans, or Sesotho need retrieval that works across the gap, which usually means testing embeddings on real queries rather than assuming. Second, POPIA: a retrieval index over customer records is a personal-information store, and needs the same access control, retention rules, and audit trail as the source system.

Brainyx AI builds these as owned [knowledge systems](https://www.brainyxai.co.za/services/enterprise-ai-solutions) rather than uploads into someone else's platform.

Mini-FAQ

A: Match the source's rate of change. Policy libraries can be scheduled; ticket data usually needs incremental updates.

A: No. Long context solves fitting, not access control, freshness, cost per query, or citation.

A: Because nothing told it not to. Handle the empty-retrieval case explicitly and return "not found in the approved sources".

Next lesson

Continue to [fluency and ownership](https://www.brainyxai.co.za/education/claude/fluency-and-ownership).

Official reference: https://docs.anthropic.com

Course hub: https://www.brainyxai.co.za/education/claude · Previous: https://www.brainyxai.co.za/education/claude/agents-tools-mcp · Next: https://www.brainyxai.co.za/education/claude/fluency-and-ownership

Return to the course hub, or explore Brainyx AI services to put the skill into a live system. Markdown: /md/education/claude/rag-vs-context.md

Book a consultation · joshua@brainyxai.co.za · Markdown mirrors