Mixedbread Quality Evals
How Mixedbread performs on public retrieval and agent benchmarks, measured against the strongest competing models.
Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.
How we measure
Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.
Toast 1, our search agent, against frontier models running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.
OfficeQA-Pro
Open OfficeQA-Pro paperHead-to-head as a standalone search agent over 89,000 pages of financial PDFs: Toast 1 vs. frontier models running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.
- 1GPT-5.6 SOL120s0.57
- 2GPT-5.6 TERRA97s0.54
- 3TOAST 1 (RRF ×3)10s0.53
- 4GPT-5.6 LUNA144s0.50
- 5OPUS 542s0.49
- 6TOAST 18s0.49
- 7KIMI K3157s0.47
- 8GLM77s0.46
- 9DEEPSEEK82s0.45
- 10QWEN148s0.43
- 11SONNET 553s0.42
- 12HAIKU 4.540s0.36
NDCG@10 vs. p50 latency per query and cost per query · higher is betterTested August 2026OfficeQA-Pro paperIntroducing Toast 1
Dataset and methodology
OfficeQA-Pro contains 89,000 pages of complex financial documents (U.S. Treasury Bulletins, dense tables, scanned PDFs) with questions that require multi-document reasoning. Here the search agent must return the ranked evidence pages for each question.
Every system runs its full agentic search loop over the same index and returns a ranking; we report NDCG@10 of that ranking, cost per query at list prices with prompt caching, and p50 latency per query. Each frontier model is swept across its reasoning-effort settings; the chart connects each model's Pareto-efficient settings. Toast 1 runs on Mixedbread Search in its default configuration and in a three-way reciprocal-rank-fusion (RRF ×3) configuration.