I once tested a system with a transaction CSV containing about ten thousand rows and asked how many times a particular name appeared. The model answered confidently. The number was wrong.
The problem was not whether the model could count in principle. Standard RAG is structurally unsuited to this task. It splits a source into chunks, retrieves a few relevant chunks, and gives those fragments to the model. That works for explaining a policy or locating a passage. It does not provide a complete view for counting, summing, finding extrema, or proving that no matching record exists.
The rule I use is simple: deterministic code owns the computation path; the model handles semantic interpretation and explains the result.
RAG returns relevant fragments, not a complete data view
Suppose a CSV becomes 200 chunks and the retriever returns the top five. Even when all five are highly relevant, they say nothing about whether the other 195 contain more matches.
Standard retrieval-based RAG therefore cannot guarantee answers to questions such as:
- How many times does this name appear?
- What is the sum of every matching amount?
- Are there no matching records anywhere in the dataset?
- What are the maximum, minimum, or average values over all rows?
Increasing top-k gives the model more fragments, but still does not guarantee a complete scan. It also raises context length and cost. Even when the full dataset fits inside the context window, a language model may still skip rows or make arithmetic mistakes in a long table.
RAG can still help with structured data. It can retrieve field definitions, business rules, and the likely dataset. Complete filtering and aggregation should be executed by a database query or a program.
Let code read the source and return a verifiable result
For this test, the tool used pandas to read the original CSV and scan every row under explicit conditions. It returned a structured result to the model, which explained the result without recalculating it.
A response can look like this:
{
"status": "exact_match",
"query": "王小明",
"rows_scanned": 10000,
"matched_rows": 7,
"source_version": "transactions-2026-08-20.csv",
"normalization": ["NFKC", "trim_whitespace"]
}This has three benefits. The computation can be repeated, the model does not need to guess whether the data view is complete, and an operator can later identify the source version and rules that produced the result.
When the data already lives in a relational database, parameterized SQL is usually the better tool. A CSV utility is useful for offline files and prototypes, but higher volume, concurrency, and update frequency introduce requirements for indexing, transaction consistency, and authorization.
Define normalization rules before matching strings
Two names that appear identical may contain full-width characters, extra whitespace, different letter case, or different Unicode sequences. Exact matching without normalization can miss records that should be grouped together.
Common steps include:
- Unicode NFKC normalization.
- Trimming leading and trailing whitespace and collapsing repeated spaces.
- Applying one case policy to Latin letters.
- Handling punctuation, company aliases, and common suffixes according to business rules.
Traditional/Simplified Chinese conversion needs more caution. Traditional-to-Simplified conversion may collapse multiple characters into one, while Simplified-to-Traditional conversion can depend on context. It may reduce some character-form differences, but it cannot guarantee that every variant is combined correctly. Keep the original value, create a separate normalized field, and make sure an audit record can return to the source text.
Normalization is business logic. It needs a version, test cases, and a change history.
Read identifiers as strings first
Account numbers, transaction IDs, and customer identifiers may contain only digits, but they are not necessarily numbers that should be calculated. If a data tool infers an integer type, leading zeros can disappear. If it infers floating point, the value may be displayed in scientific notation or lose precision.
Read identifier fields as strings. Parse only genuine measures such as amounts and quantities into numeric types.
Financial number parsing also needs explicit handling for thousands separators, currency symbols, full-width digits, nulls, and accounting negatives such as (1,250). A failed parse should not silently become zero. Return the number of invalid rows so data-quality problems do not disappear inside a total.
Distinguish exact matches, possible matches, and no match
Name matching is not always binary. Exact matching can miss spelling differences, aliases, or suffixes. A loose fuzzy threshold can merge different entities.
For workflows that require review, use at least three states:
- Exact match: the approved normalization rules are sufficient to identify it.
- Possible match: similarity or some fields are close, but a person needs to review it.
- Not found: no match exists within the stated source version, fields, and rules.
The third state needs scope. Record how many rows were scanned, which fields were used, and which data version was checked. The tool can say “not found in this complete scan.” It cannot conclude that the entity does not exist in the real world.
This classification can assist with sanctions or customer-name screening, but it is not a replacement for a formal AML or KYC system. Production screening may also need aliases, romanization, date of birth, nationality, corporate relationships, risk rules, and authorized review.
A prompt cannot replace tool-layer enforcement
A system prompt can tell the model to call a tool whenever it sees a counting question. That instruction is not an execution guarantee. A stronger application identifies the intent, routes it to the approved tool, and limits the final answer to the tool result.
High-risk tools can add controls such as:
- Accept only approved fields and operations.
- Select the file or table on the server instead of accepting an arbitrary path.
- Use parameterized SQL and a least-privilege database account.
- Return the data version, scan scope, and invalid-row count.
- If execution fails, report the failure instead of estimating an answer.
The model can interpret what the user wants to calculate. The program guarantees that the complete dataset is processed consistently.
An audit log does not need to copy sensitive content
Traceable computation does not require logging the entire dataset or every raw query. Useful fields include:
- Tool and rule versions.
- A data-source identifier or hash.
- Execution time, request ID, and authorized principal.
- Fields used, operation type, and normalization rules.
- Rows scanned, rows matched, invalid rows, and execution status.
Sensitive query values can be masked, tokenized, or hashed under controlled conditions, depending on the audit requirement. Access to the logs should be restricted. The design needs both traceability and data minimization.
Divide responsibility among RAG, code, and the model
I separate the system into three layers:
- RAG: retrieve relevant documents, rules, and field definitions.
- Deterministic code: execute complete filtering, counting, aggregation, and sorting.
- LLM: interpret natural-language intent, choose an approved operation, and explain the result and its limits.
This does not remove the model. It keeps the model focused on the part it handles better. Whenever an answer must follow the same rule and produce the same result on every run, put that path into code that can be tested, repeated, and audited.
If an existing RAG system answers counting or aggregation questions, sample those requests and check how much of the source the model actually received. If the input is not a complete data view, route the calculation to a database or program.
Frequently asked questions
If the complete CSV fits in context, can the model count accurately?
Not reliably. Full context removes retrieval omissions, but a model can still skip rows or calculate incorrectly in a long table. Use a database or program for exact counts and let the model explain the result.
Which tasks fit “compute the path, prompt the judgment”?
It applies to filtering, aggregation, date ranges, path search, permission checks, and rule validation when the result must be deterministic. A model may assist with semantic classification or fuzzy matching, but high-impact conclusions still need explicit rules or human review.
Can this tool be used directly for AML name screening?
It should not replace a production AML system. Simple exact and fuzzy matching can assist a workflow, but formal screening also needs multi-field matching, list versions, risk rules, human review, appeal or remediation paths, and complete audit records.
Sources and further reading
About the author
KJ Huang (KJH) is a Taiwanese software engineer and technical leader with more than eight years of experience across AI, blockchain, gamification, cybersecurity, and finance. His current work focuses on enterprise self-hosted LLM platforms, RAG, and AI governance. More at kjhuang.com.
This article is part of the AI Engineering in Banking series.
Also in this series: Why Banks Self-Host LLMs and RAG Optimization in Practice.
Chinese version: RAG 為什麼不適合回答「有幾筆」？.