Benchmark report · Updated September 2, 2026
Financial Agent Benchmark
Financial answers should survive inspection. On the Vals Finance Agent public set, Drillr MCP scores 94.06%, against 76.30% for Claude Opus 4.6 with built-in web search. In a controlled run over 200 questions, the same model scored 80.76% with Drillr MCP and 33.86% with web search, using 55% fewer tool calls.
01 · Why a new benchmark
Search returns passages. Financial questions need context.
Ordinary search tools return passages that resemble the question. The passage arrives without the table it sat in, the period it belongs to, or the filing that later corrected it. That loss of context is the biggest weakness of search-based agents.
Existing benchmarks do not measure it. When the answer sits in one passage, a system that matched the words scores the same as one that understood the filing.
The Drillr Benchmark is built to tell them apart. Every question requires the full context: which entity, which period, which version of the figure, and how figures across documents relate.
02 · How questions are made
Detectors find the seed. Code does the math. People sign off.
Seed
Detectors scan as-reported financial data for traps. Each hit becomes a seed after review. Every seed keeps the query that reproduces its true value.
Compose
Code does all the math. The language model only writes the question text and the scoring items.
Validate
Every number in the question, answer or scoring items must match the source data. A made-up figure cannot pass.
Review
Two people review each question before it enters the set.
03 · Task types
Nine kinds of question. Each is built around one way to be wrong.
Eight task types are Drillr’s own. Each has a detector that looks for one kind of trap in as-reported data. The ninth pools questions from other public benchmarks.
| Task type | What it tests |
|---|---|
| EPS numerator | The numerator of basic EPS. It appears only in the EPS note, next to more visible figures that look right but are not. |
| Cross-entity breadth | Exact footnote values from several companies. Each has its own fiscal year-end, units and row labels. |
| Company Search | Name the companies that match a description. Scored by how many of the targets are found. |
| Event base rate | How often a kind of corporate event happens. This has to be counted across many filings. |
| Footnote nonstandard | Nearby values that share the same keywords but belong to a different scope, period, unit, row or column. |
| Keyword collision | The same words used for different entities, dimensions or metrics. |
| Cross-family revision | A figure that was later corrected. The question does not say so. The old value is still on news sites and aggregators. Exact match is required. |
| Restatement | Which version of a figure is current across formal disclosures. The old value is real and kept as a trap. |
| Other public benchmarks | Questions from Finance Agent v2, Daloopa, FinSearchComp and BigFinanceBench. Each was re-checked against as-reported data before it was kept. |
04 · Scoring
The judge says pass or fail. The engine does the math.
Pass or fail per item
The judge marks each scoring item pass or fail. It does not give points. The engine adds up the weights. If an item depends on one that failed, it scores zero too.
Two passes, lower score kept
Every answer is judged twice. The lower score counts. Any disagreement goes to a person.
Tolerance set per question
Numeric answers allow 3% error, unless the question asks for an exact match. Thousands, millions and billions are treated as equal. Rounded or nearby values are not.
Every question worth 10
Scores are normalised so every question counts the same. Correct means full marks. Full, partial and zero are reported separately.
Company Search is deterministic
List questions skip the judge. The score is the share of target companies named. Extra companies cost nothing.
Ground truth cites the filing
Every answer records the filing type, date, section and source text. A disputed score can be traced back.
05 · Running a configuration
Same model, same prompt, same budget. Only the tools change.
| Model | Same model in every run |
|---|---|
| Prompt | One shared system prompt |
| Temperature | 0 |
| Agent turns | 20 per question |
| Tool calls | 20 per question |
| Retrieval | The only change: web search or Drillr MCP |
A run answers every question once with one setup. The agent can use a tool only while it has budget left. Every call is logged: the tool, the arguments, the latency, the size of the result and whether it was cut short. That is how tool use is compared alongside accuracy.