# RAG With Citations: How to Make a RAG System Cite Its Sources

*Give each passage an address when you index it. Let the model cite only the passage IDs it was given. Then check each sentence against its passage. Cut what no page backs up. The citation comes from the search. The model never gets to write one.*

By Cameron Sullivan, Founder & CEO, Techify AI · October 15, 2026 · 10 min read

> **Key takeaways**
>
> - A citation you can trust is set when you index, not when you answer. Each chunk carries its document, version, page and passage. The model may cite only the chunk IDs it was given.
> - Search two ways, by exact words and by meaning, then rerank. Anthropic tested this in 2024. Adding keyword search to embeddings cut failed retrievals by 49%. A reranker on top cut them by 67%.
> - Check each sentence against the passage it cites. The 2023 ALCE benchmark tested this. Even the best models lacked full citation support 50% of the time. That was on one of its test sets.
> - Let the system decline. A line that no page backs stays out. We call the rule No Page, No Answer.
> - Test with 15 to 25 real questions. Each one comes with the passage a right answer must cite. Run the set again after each change.

RAG stands for retrieval-augmented generation. The system finds passages in your own docs. Then a model writes its answer from them. A RAG system cites its sources well when the citation is plumbing, not prose.

Here is how that works. You split each doc into passages. Each one is stored with its address: doc, version, page and paragraph.

A question comes in. The system finds the best passages. It hands them to the model under short IDs. The model writes its answer. It tags each sentence with the IDs it used. Then code, not the model, turns those IDs into links that open the page.

A second check reads each sentence next to its passage. It cuts or flags what the passage doesn't say. And if nothing in your docs backs an answer, the system says so.

Five moves. Plenty of systems sold as "cites its sources" make two of them.

## Why do RAG systems cite the wrong source?

Because the usual setup asks the model to do the citing. You paste five passages into a prompt. You add "cite your sources." The model does as it's told. It writes what looks like a reference. Often it's the right one.

Often is the problem. ALCE is a 2023 benchmark, a shared test built for this exact question. Even the best models lacked full citation support 50% of the time. That was on one of its test sets. So about every other answer held a claim its own sources didn't back.

The failures come in a few shapes. Each has its own fix.

| What goes wrong | What the reader sees | The fix |
|---|---|---|
| The model writes the reference | A title or page that looks real and doesn't exist | The model cites IDs; code builds the link |
| The citation points at a whole file | A 60-page PDF, and good luck | Store the page and passage on every chunk |
| The chunk was cut mid-argument | A quote that means the opposite out of context | Cut at headings and paragraphs, with some overlap |
| Two versions of one document | Last year's answer, cited correctly | Store the version and date; fetch the current one |
| A related passage, not a supporting one | A real page that doesn't say what the line says | Check each sentence against its passage |
| Nothing useful came back | A sure-sounding answer from the model's training | Score the answer and let it decline |

The last two rows do the most harm. The citation is real. It opens. It just doesn't back the line beside it. And a reviewer who sees a working link stops reading.

## Step 1: Give every passage an address when you index it

Indexing is when the system reads your docs and stores them for search. That is where citations are set, before the first question comes in.

Each stored chunk is a short passage. It needs labels that lead a person back to the exact spot. Those labels are its metadata:

- a stable doc ID, and the version or the date it took effect
- the title as your readers know it
- the page, and the heading the passage sits under
- where the passage sits in the source, so a viewer can highlight it
- who may see it, if not everyone may

Then cut along the doc's own shape. Cut at headings and paragraphs. Keep a table in one piece with its header row.

Let next-door chunks overlap a little. Then a sentence that leans on the one before it isn't cut off.

Scanned PDFs need OCR first. OCR turns a picture of a page into text. Skip it, and the system will cite pages it can't read.

We built an assistant for a small HR team. It answers questions on employment standards. Each doc is taken apart with its headings and tables kept in place. Then it's cut into chunks along that same shape.

Names and other private details come out first. Then the text goes into the index. The order matters. Names taken out after indexing are already in the index.

## Step 2: Search on two paths, then rerank

Embeddings turn text into numbers that hold its meaning. A search on them finds passages by meaning. Ask about "lenders," and it finds a passage that only ever says "banks."

But embeddings are weak on exact strings. Think of a ticker, a clause number, an error code or a part number. Keyword search is strong right there. (BM25 is the usual method.)

So run both and merge the two lists. Then rerank the list. A reranker is a model trained to score one passage against one question.

Anthropic shared numbers on this in September 2024. It was testing a method it calls contextual retrieval. A failed retrieval is a search that misses the right passage.

Adding keyword search to its embeddings cut those misses by 49%. Adding a reranker cut them by 67%, from 5.7% of queries to 1.9%. That was measured on the top 20 chunks.

Then hand the model fewer passages than feels safe. A 2023 study of long inputs is called "Lost in the Middle." It found that models do best when the key text comes first or last. They do clearly worse when it sits in the middle.

The reranker's job is to put the few passages that matter first. That way the twentieth never gets a chance to be cited.

We built a research copilot for a financial publisher. It works this way over 10,000+ of the publisher's own docs. One path reads a short summary of each doc first. Then it digs into the passages of the best docs. The other path goes straight to the passages.

Both lists are merged. Then a cross-encoder reranks them, before the model sees a word. It's a reranker that reads the question and each passage side by side.

## Step 3: Make the model cite IDs, not write references

In the prompt, give each passage a short label, S1 to S5. Ask for structured output. That means a fixed format your code can read.

The answer comes back as a list of sentences. Each sentence lists the IDs that back it. It also gives the exact words it relied on. Then your code holds the model to that format:

- An ID that wasn't in the set is rejected. The model can't cite a document it never saw.
- A factual sentence with no ID is cut, or shown as unsupported. Your rule decides which.
- The display code looks up each ID in the stored labels. It builds the title, page and link. The model never types a title, a page number or a URL.

That's most of the trick. It isn't fancy. The model writes the sentence. The search decides where the sentence came from. A model that can't type a reference can't make one up.

Cite per sentence, not per answer. Five sources at the bottom tell the reader the answer is in there somewhere. They don't say which line came from where. That's the only question a reviewer asks.

## Step 4: Check every sentence against the passage it cites

A real citation can still be wrong. So once the answer is written, run a check. Read each sentence next to the passage it cites. Ask one narrow question: does this passage back this sentence?

The question is narrow. So a smaller, cheaper model can handle it most of the time. The check adds little to the bill.

There are three outcomes. Supported, and the sentence stays. Partly supported, and it's rewritten to what the passage says, or flagged. Not supported, and it's cut.

Sometimes the cuts leave nothing. Then the system says your docs don't cover the question. We call the rule No Page, No Answer. A line no page backs has nothing to cite. So it stays out.

A system built on No Page, No Answer will sometimes say less than a chatbot. For a publisher, a compliance team or an HR desk, saying less is the point.

## What should a cited answer look like on screen?

A reader must be able to check a line in one click. If not, no one will check it. A good cited answer shows six things:

1. A marker after each sentence, numbered in the order its sources first appear.
2. A source list with the title the reader knows, the page, and the version or date.
3. The quoted words, so the reader sees what each sentence rests on.
4. A link that opens the document at that page, with the passage lit up.
5. How well the sources back the answer, as a score or in plain words.
6. Any disclaimer you need, added by code as a fixed block.

The last item earns a paragraph. Some answers need a disclaimer. Think of market notes, legal facts or anything clinical. Don't ask the model to add it.

Tell a model to add it, and it will add it most of the time. Most of the time is not a compliance standard. So add it in code after the answer is written, on every output.

In the publisher's build, code adds a required disclaimer to each answer. That covers four business units and a team of 50 editors. It costs no model call.

## When should a RAG system refuse to answer?

When the search comes back weak. Or when the check cuts most of the draft. Put a number on both. Score each answer before a person sees it. Send the low scores to a person, not to the screen.

The HR tool above scores each answer in three parts:

- 80% retrieval similarity: how well the passages found match the question
- 10% source diversity: whether the answer rests on more than one source
- 10% response completeness: whether each part of the question got an answer

The heavy weight on retrieval is on purpose. Take a fluent, complete answer built on a weak match. It can gain at most 0.20 from the other two parts. So it can't write its way over the line.

Below the line, a person on the HR team gets the question. The whole chat comes with it. (How that line gets set is in our guide to [AI for HR](/insights/ai-for-hr).)

Keep the declined questions. A list of "our docs don't cover this" is the best map of your gaps. The publisher's build has an admin screen for this. It holds a queue of unanswered questions for the editors to review.

## When not to build RAG with citations

Your set of docs is thin. Under a couple of hundred docs, use a good search box and a tagging habit. They cost less and find things about as fast.

The answers aren't written down. Then a system that follows No Page, No Answer will decline most questions. That's correct, and a poor use of money.

No one owns the docs. One policy has three versions and no dates. A third of it sits in someone's inbox. The system will cite the wrong version with a straight face. Fix who owns what first.

The sources aren't yours. Answering from someone else's research or news feeds is a license question first. The engineering comes second.

## How do you test citation accuracy?

With your own questions, never a vendor's. Take 15 to 25 questions people really asked last month. For each one, write down the passage a right answer has to cite. Add a few that your documents can't answer. Then check four things on each run.

| Check | The question it answers |
|---|---|
| Retrieval | Did the right passage come back near the top? |
| Citation precision | Does each cited passage back the sentence it's on? |
| Citation coverage | Does every factual sentence have a citation? |
| Declines | Did it decline the questions your documents can't answer? |

Run the set again after each change. That means new docs, a new chunking rule, a new model or a new prompt.

The HR assistant ships with exactly this kind of set. It holds 15 to 25 hand-picked questions, each with the passages it should cite. The checks go on after launch.

And trace each model call. We use OpenTelemetry and Langfuse, tools that log each step. Say a link goes wrong in month three. Then "which step did it?" is something you look up.

Worried about cost? The extra steps are cheap next to the answer itself. Reranking and the sentence check run on small models. Questions that repeat can be cached, which means saved and reused.

Our guide to [how much it costs to run an AI system](/insights/cost-to-run-an-ai-system) has the full bill.

## FAQ

### What is RAG with citations?

RAG stands for retrieval-augmented generation. With citations, each statement links to the passage it came from. The system finds passages in your own docs. The model writes only from them. It tags each sentence with a passage ID. Code turns the IDs into links that open the source page. A check afterward makes sure each cited passage backs its sentence.

### How do you stop a RAG system from making up citations?

Don't let the model write references at all. Give each passage it finds a short ID. Make the model cite only those IDs, in structured output. Reject any ID that wasn't in the set. Have code build the title, page and link from the stored metadata. Then check each sentence against its passage. A real citation can still fail to back the line.

### What is hybrid search in RAG?

Hybrid search runs two searches and merges them. Keyword search finds exact strings, such as tickers, clause numbers or error codes. Embedding search finds meaning. A reranker then puts the merged list in order. Anthropic tested this in 2024. Using both cut failed retrievals by 49% against embeddings alone. Adding a reranker cut them by 67%.

### Should a RAG system answer when it isn't sure?

No. Score each answer on how well the passages found back it up. Below a set line, decline or send the question to a person. One HR assistant we built puts 80% of the score on retrieval similarity. So a fluent answer on a weak match can't score its way through. Keep the declined questions. They show where your documents have gaps.

### Can AI add a compliance disclaimer to every answer?

Yes, and it's reliable only if code adds it. Add the disclaimer as a fixed block after the answer is written. Do it on every output. Don't ask the model to include it. We built a research copilot for a financial publisher. There, the disclaimer goes on every answer across four business units. It adds no model call and no wait.

## Sources

- Gao, Yen, Yu and Chen, "Enabling Large Language Models to Generate Text with Citations" (the ALCE benchmark), EMNLP 2023: [arxiv.org](https://arxiv.org/abs/2305.14627)
- Anthropic, "Introducing Contextual Retrieval", September 19, 2024, for the 49% and 67% reductions in failed retrievals (top 20 chunks): [anthropic.com](https://www.anthropic.com/news/contextual-retrieval)
- Liu et al., "Lost in the Middle: How Language Models Use Long Contexts", 2023, published in Transactions of the Association for Computational Linguistics: [arxiv.org](https://arxiv.org/abs/2307.03172)
- Techify build library: a research copilot for a financial publisher, and an employment-standards assistant for an HR team. Client names withheld. [The range of builds](/work)

## Related

- [AI for financial publishers and research firms](/insights/ai-for-financial-publishers)
- [AI for HR: cited answers and the line where a person takes over](/insights/ai-for-hr)
- [HIPAA compliant AI: what the phrase should mean](/insights/hipaa-compliant-ai)