PolicyMind AI
A policy assistant that answers across a corpus of PDFs and cites the page.
Answers questions across eight company policy PDFs — 80 pages — and cites the document and page behind every claim. The part worth pointing at is not that it answers: it is that a relevance floor lets it return nothing, so “the documents don’t specify that” is a real output rather than a failure mode.
- Hybrid retrieval — dense vectors in ChromaDB and BM25 keyword search over the same chunks, fused by Reciprocal Rank Fusion rather than a weighted score blend.
- A cross-encoder reranks the fused top 10 down to 5; anything scoring under the floor is dropped before the model ever sees it.
- Follow-ups are rewritten into standalone questions first, so “what about contractors?” retrieves something instead of nothing.
- Citations come from chunk metadata, not from asking the LLM where it read something.
The problem
A policy assistant that always produces an answer is worse than one that sometimes declines, because the person asking cannot tell the two cases apart. Everything hard about this project is on the retrieval side: pure embedding search is strong on paraphrase and weak on the exact tokens policy documents turn on — a clause number, an acronym, a defined term — and by the time weak context reaches the model, the model will use it. So the design question was never “which LLM” but “what is allowed to reach it”.
Constraints
- Eight policy PDFs, 80 pages. Small enough that I can tell a wrong answer from a right one by hand, which is what makes it a workable corpus to build against.
- Free Gemini tier — cost is per request, so the filtering has to happen in retrieval rather than by stuffing a large context and hoping.
- Runs locally on CPU. Embeddings, BM25 and the cross-encoder all have to be small enough to be interactive without a GPU.
How it fits together
8 policy PDFs
80 pages
Chunk per page
50 words, 10 overlap
Embed
all-MiniLM-L6-v2
ChromaDB
persistent, page metadata
Dense + BM25 → RRF
top 10
Cross-encoder rerank
top 5, score floor
Gemini 2.5 Flash
grounded prompt
Answer + citations
document and page
Decisions and what they cost
Every one of these had a cheaper or more obvious alternative. These are the reasons I didn't take them.
Hybrid dense + BM25 retrieval, fused with Reciprocal Rank Fusion.
Instead ofDense vectors alone, or a weighted blend of cosine similarity and BM25 score.
Embeddings are good at paraphrase and bad at exact tokens — a clause number or a defined term is semantically uninformative but lexically decisive, which is precisely what BM25 is good at. The two are complementary rather than redundant. RRF fuses by rank instead of score on purpose: cosine similarity and BM25 scores live on different, unbounded scales, so any weighted blend needs its weights retuned whenever the corpus changes. Rank has no units, so it needs no tuning.
Rerank with a cross-encoder instead of trusting the retriever's order.
Instead ofRaise top-k and let the model sort out what is relevant from a larger context.
A bi-encoder embeds the question and the chunk separately, so it can only ask whether they are about the same topic. A cross-encoder reads both together and can ask whether this chunk answers this question, which is a different and harder question — and the one that matters. It is far too expensive to run over the whole corpus, which is exactly why it belongs after retrieval rather than instead of it: 10 pairs is affordable, 10,000 would not be.
A hard relevance floor that is allowed to return zero chunks.
Instead ofAlways pass the top 5 through, whatever they scored.
The failure mode of a policy assistant is not a missing answer, it is a fluent wrong one — someone acts on it. If the best chunk available is weak, passing it anyway hands the model material it will dutifully use. Dropping it means the honest answer survives to the user. This is the one decision in the project I would defend hardest, because it makes the demo look worse and the system behave better.
Rewrite follow-up questions into standalone ones before retrieval.
Instead ofEmbed the raw follow-up, or concatenate the whole conversation into the query.
“What about contractors?” carries almost no retrievable signal by itself — the noun it depends on is three turns back. Concatenating the history instead drags every earlier topic into the query vector and blurs it. One extra call that resolves the pronouns turns an unretrievable question into a retrievable one, and it only runs when there is history to resolve.
What broke
Chunks are 50 words with a 10-word overlap, cut inside each page. That is precise for retrieval and wrong for policy prose: a rule and its exception routinely land in different chunks, and retrieving only the first gives an answer that is correctly cited and still wrong.
FixPartly covered by returning up to five chunks and instructing the model to combine them rather than answer from one. That reduces the failure, it does not remove it — the real fix is structure-aware chunking that splits on clause boundaries instead of a word count, and I have not built it.
Reranked results were repeatedly several chunks off the same page, which crowded the other seven documents out of the context entirely.
FixDeduplicate on (source, page) after reranking and before truncating to five, so the context spans pages rather than restating one.
The relevance floor is a magic number. It sits on the cross-encoder's raw logit scale, which has no calibrated meaning, and I picked it by watching scores on questions I already knew the answers to.
FixUnresolved, and named here rather than hidden: it is tuned to this corpus and this reranker, and changing either invalidates it. Calibrating it properly needs a labelled question set, which is the same thing missing from the measurements below.
Results
- Corpus
- 8 PDFs
- Retrieval
- Hybrid
- Reranker
- Cross-encoder
- Context sent
- ≤ 5 chunks
What I'd do next
The evaluation set, before anything else. Thirty questions with known answers and known source pages would convert most of this page from reasoning into evidence, and would let the relevance floor be chosen rather than guessed. After that, chunking on clause boundaries instead of a fixed word count — it is the largest remaining source of confidently-cited wrong answers, and no amount of reranking fixes a chunk that was split in the wrong place.