Lab / rag-strategy-bench
RAG Strategy Bench
Side-by-side evaluation of chunk sizes, embedding models and retrieval strategies on the same document set — the harness that picked Kilele Archade's retrieval settings.
What it tests
RAG demos look identical until you measure them. Picking chunk size or an embedding model by intuition means guessing with a production system on the line.
How it works
One fixed question set with known-good source passages. The harness runs every configuration — chunk sizes, overlap, embedding model, top-k, metadata pre-filtering, re-ranking — through the same corpus and scores retrieval hits and answer faithfulness.
Results land in PostgreSQL so each run is queryable: retrieval settings become a table you can diff instead of a Slack thread of vibes.
What came out of it
The retrieval configuration shipped in Kilele Archade came from this bench: metadata-filtered pgvector search with re-ranking beat naive top-k on the same corpus. It also proved that improving retrieval is mostly an engineering task, not a bigger-model task.