RAG Optimization: Measure First, Then Tune Retrieval

When I started working on the bank's internal RAG system, the first problem was not an outdated embedding model. It was that nobody could answer a basic question: how accurate is the system now?

The embedding model, chunk size, and top-k settings were mostly defaults. Changes were judged with a few hand-picked questions rather than a fixed test set. In that situation, replacing a model produces impressions, not evidence. A few better answers do not prove that retrieval improved across the workload.

My tuning sequence is now fixed: establish an eval set and a baseline, then change one variable at a time. The sections below cover the parts that have mattered most for Chinese financial documents, along with the limits that need to remain visible.

Build a repeatable eval set first

The first eval set does not need to be large, but it should come from real work. Start with 50 to 200 common queries. For each query, label one or more relevant documents, then run the same questions against a fixed version of the knowledge base.

At the retrieval stage, I track at least two metrics:

Recall@k: whether the top k results contain a document that should have been retrieved.
MRR: how early the first relevant document appears.

Recall@k answers “did retrieval find it?” MRR shows whether the correct document is consistently buried behind weaker matches. If the system also generates an answer, evaluate answer correctness, citation quality, and refusal behavior separately. Retrieval metrics do not describe the whole RAG system.

The first evaluation script can be simple. What matters is recording the question set, document version, model, and parameters so that every change has a comparable result. In a financial environment, those records also explain why a model or setting changed instead of leaving only “it seemed better.”

Verify the embedding model is being used correctly

Selecting the right model does not guarantee that the integration is correct. The multilingual-e5-large model card, for example, says that retrieval queries and passages should use the query: and passage: prefixes. It also truncates long input at 512 tokens. Missing the prefixes or ignoring the input limit can reduce retrieval quality.

Before replacing a model, check:

Whether queries and documents use the required prefix or instruction.
Whether pooling and vector normalization match the model card.
Whether real inputs exceed the model's token limit.
Whether indexing and query encoding use the same model and preprocessing.

We also evaluated BGE-M3. It supports multiple languages, inputs up to 8,192 tokens, and dense, sparse, and multi-vector retrieval. A longer input limit still does not mean that an entire document should become one vector. Chunk length needs to be evaluated with the institution's own documents and queries.

After changing an embedding model, old vectors cannot be reused. Documents and queries must be encoded with the same model and preprocessing pipeline. Rebuild the index so both sides are in the same vector space.

Base chunking on tokens and document structure

Chinese character count is not the same as token count. The ratio changes with the tokenizer, punctuation, content, and mixed Chinese-English text. A character-based splitter may produce chunks that the embedding model silently truncates.

Use the target model's tokenizer to measure length, and preserve headings, sections, and tables where possible. One starting point I used was roughly 600 tokens with an 80-token overlap, but that was only a baseline for one document set.

Small chunks can separate a rule from its context. Large chunks can mix several topics into one vector and crowd out the most relevant passage. A useful experiment is to compare 300, 600, and 900-token chunks while holding other variables constant, then measure Recall@k, MRR, and latency.

For policies and operating procedures, avoid separating a heading from its body. Even if the heading is not repeated in the text, include the document name and heading hierarchy in chunk metadata or the retrieval text so the model knows where the passage belongs.

A reranker is often better than increasing top-k

Vector search is efficient at producing a candidate set, but the first few positions may be weak when provisions use similar language. A two-stage pipeline can retrieve a broader set, such as 15 candidates, then use a cross-encoder reranker and pass the best five to the generation model.

I have used bge-reranker-v2-m3. On the dataset available at the time, it improved precision more than increasing top-k alone. Candidate count, retained count, GPU memory, and latency still depend on hardware, document length, and concurrency. A number from one deployment should not become a universal setting.

A reranker cannot recover a document that the first stage never retrieved. If Recall@15 is already low, inspect the embedding model, chunking, query rewriting, or hybrid retrieval before tuning the reranker.

Chinese BM25 needs an appropriate tokenizer

BM25 is valuable for regulation numbers, product names, identifiers, and exact terms, but many default implementations tokenize on spaces. Chinese sentences do not contain natural spaces, so a whole sentence may be treated as one token.

Whether BM25 is worth adding should be decided by the eval set. If vector retrieval is weak on codes, clause numbers, or rare terms, add Chinese word segmentation or character n-grams and combine the score with dense retrieval. If the workload is mostly semantic questions, the operational cost of hybrid retrieval may not produce the same benefit.

I treat BM25 as a tool for a demonstrated lexical matching gap, not as a mandatory RAG component.

OCR quality sets the ceiling for retrieval

If a scanned PDF, table, or two-column document is parsed incorrectly during ingestion, neither embeddings nor rerankers can recover text that is missing.

Document-processing tools such as Docling can combine OCR, layout analysis, and table extraction, but the output still needs sampling and review. Financial documents are affected by small type, stamps, cross-page tables, headers, and footers. In addition to overall character accuracy, check heading levels, clause numbers, dates, amounts, and table fields.

For important collections, add ingestion gates. Abnormally low text volume, poor OCR confidence, or failed table parsing should trigger review instead of sending the file directly into the production index.

Do not embed page numbers; keep them in metadata

Even with accurate OCR, page numbers and source locations are harder to manage than filenames when RAG needs to produce citations. embedding(chunk.text) returns a vector. It does not automatically carry the source file, page, or coordinates.

The problem is not that a vector database cannot store page numbers. Page location and semantic content are different kinds of information. Content belongs in semantic search; source location belongs in metadata.

Adding “page 37” to the chunk before embedding does not create reliable page lookup. The number has little semantic meaning, and many documents have a page 37. When a user asks for a specific page, the request contains a structured condition that should not be left to vector similarity.

Several page concepts are easy to confuse:

PDF page index: the position of a page object in the PDF, which may not match the viewer label.
Printed page label: the number printed in the footer, such as 1, 2, 3 or i, ii, iii.
Chunk page span: every page touched by one chunk.

Covers, tables of contents, appendices, and blank pages can make the PDF page index differ from the printed label. Adding one page to a later document version also shifts all subsequent indices. A single page_number field is therefore ambiguous unless its numbering system and document version are defined.

Printed labels are not always available as structured PDF metadata. Some exist only as footer text; a scanned document may contain only an image. If a parser removes headers and footers as noise, the number may disappear before chunking. When the printed label is required for citations, the ingestion pipeline must extract and validate it separately.

Chunking introduces another problem. A paragraph or table may cross two pages. A token-aware chunker may also merge adjacent content under one heading. If the pipeline flattens a PDF to Markdown and then chunks plain text, page anchors, document item references, and bounding boxes may be lost.

I keep provenance metadata with each chunk instead of treating page numbers as embedding text:

{
  "document_id": "credit-policy",
  "document_version": "2026-08-20",
  "source_hash": "...",
  "chunk_id": "credit-policy-0042",
  "doc_item_refs": ["#/texts/128", "#/tables/7"],
  "page_anchors": [
    {
      "pdf_page_index": 40,
      "printed_page_label": "36",
      "bbox_xyxy_pt": [90, 280, 506, 306],
      "bbox_origin": "top-left"
    },
    {
      "pdf_page_index": 41,
      "printed_page_label": "37",
      "bbox_xyxy_pt": [88, 70, 510, 210],
      "bbox_origin": "top-left"
    }
  ]
}

The exact schema can differ, but it should answer three questions: which document version produced the chunk, which pages it spans, and whether the UI can return to the source location. Define whether pdf_page_index is zero-based or one-based. Define the bounding-box coordinate system, unit, origin, and structure for multiple regions on one page. If the UI highlights a citation, a page number alone is not enough; keep a bounding box or an equivalent source anchor.

The query path should also split. General semantic questions use vector search, then build the citation from metadata. Requests for a page, chapter, or document version should become filters over fields such as document_id, document_version, and page data. A vector database such as Qdrant can store these fields as payload and combine them with a vector query, but only if ingestion preserved the provenance.

When the two numbering systems differ, show both: “printed page 37 (PDF page 41).” That is easier to verify and less likely to drift silently after a document revision.

Keep cross-language and Chinese normalization consistent

Multilingual embeddings can support cross-language retrieval, but performance differs by model and domain. Product names, regulatory terms, and Taiwanese financial language still need to appear in the eval set.

If the knowledge base contains Traditional and Simplified Chinese, apply the same normalization policy to indexing and query processing. OpenCC can create an additional retrieval field, but conversion may be one-to-many or context-dependent. Keep the source text unchanged and use the normalized form only for search or matching.

Small document sets may not need RAG, but long context is not a guarantee

If the knowledge scope is one short document that fits reliably in context, passing the full text may be simpler than building a retrieval pipeline. It removes one failure mode: the relevant passage was never retrieved.

Long context does not guarantee that the model uses every passage correctly. Test lost-in-the-middle behavior, citation quality, refusals, and cost. Choose between RAG and full-context input by comparing task-level results, not only the advertised context-window size.

Change one variable at a time

A RAG pipeline includes parsing, chunking, embeddings, indexing, query rewriting, hybrid search, reranking, and generation. If three components change at once, neither improvement nor regression can be attributed reliably.

My usual sequence is:

Fix the eval questions and document version.
Verify the embedding prefixes, length limits, and normalization.
Compare embedding models and rebuild the index when needed.
Tune chunk size, overlap, and document structure.
Add a reranker and measure quality, latency, and resource use.
Evaluate Chinese BM25 for lexical retrieval gaps.
Confirm that PDF page index, printed page label, chunk page span, and the source anchors required to return to the original location are stored in metadata.
Add OCR and cross-language issues to ingestion quality checks.

Keep the configuration, metrics, and rollback path for every step. That makes each change reviewable and easier to operate in an environment that requires long-term auditability.

If you are preparing to replace an embedding model or reranker, start by freezing 50 real queries and running the current baseline. Even without changing a component, that result makes the next decision easier.

Frequently asked questions

How large should the first RAG eval set be?

Start with 50 to 200 real questions that cover the main document types, common wording, and confusing cases. Annotation quality and version control matter more than the raw number. Expand the set as new failures appear.

Must the index be rebuilt after changing the embedding model?

Yes. Different embedding models usually produce incompatible vector spaces. Documents and queries must be encoded with the same model and preprocessing pipeline.

Can OpenCC solve every Traditional/Simplified Chinese retrieval issue?

No. It reduces character-form differences, but terminology, context, and regional usage still vary. Preserve the original text, apply a consistent policy on both sides, and measure the result with real queries.

Sources and version notes

About the author

KJ Huang (KJH) is a Taiwanese software engineer and technical leader with more than eight years of experience across AI, blockchain, gamification, cybersecurity, and finance. His current work focuses on enterprise self-hosted LLM platforms, RAG, and AI governance. More at kjhuang.com.

This article is part of the AI Engineering in Banking series.

Also in this series: Why Banks Self-Host LLMs and Why RAG Can't Answer "How Many?".

Chinese version: RAG 優化實戰：先建立評測，再調整檢索管線.

RAG Optimization in Practice: Measure First, Then Tune Retrieval