Mixedbread Quality Evals
How Mixedbread performs on public retrieval and agent benchmarks, measured against the strongest competing models.
Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.
How we measure
Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.
Toast 1, our search agent, against frontier models running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.
OfficeQA-Pro V2
Open OfficeQA Pro V2 (Databricks)End-to-end task result: GPT-5.6 Sol in Codex with Toast 1 as its search sub-agent, vs. every system in Databricks' OfficeQA Pro V2 release (Genie and vendor harnesses). Higher answer correctness at lower cost per task is better.
- 1GPT-5.6 SOL HIGH (CODEX + TOAST 1)$1.1870%
- 2GPT-5.6 SOL LOW (CODEX + TOAST 1)$0.7561%
- 3CLAUDE FABLE 5 (GENIE)$4.2660%
- 4GPT-5.6 SOL (GENIE)$8.1758.6%
- 5GPT-5.6 TERRA (GENIE)$1.6753.4%
- 6CLAUDE FABLE 5 (VENDOR HARNESS)$38.0045%
- 7KIMI K3 (GENIE)$2.3540%
- 8GLM 5.2 (GENIE)$3.1340%
- 9SONNET 5 (GENIE)$5.5140%
- 10GPT-5.6 LUNA (GENIE)$0.6038.8%
- 11GPT-5.6 SOL (VENDOR HARNESS)$4.6833.3%
- 12GPT-5.6 TERRA (VENDOR HARNESS)$1.2621.2%
- 13SONNET 5 (VENDOR HARNESS)$5.0015.7%
- 14GPT-5.6 LUNA (VENDOR HARNESS)$0.1215%
Answer correctness vs. cost per rollout · higher is betterTested August 2026OfficeQA Pro V2 (Databricks)Introducing Toast 1
Dataset and methodology
OfficeQA Pro V2, released by Databricks, evaluates answer correctness on 90 questions set in realistic, complex enterprise financial situations over large document collections.
GPT-5.6 Sol runs inside OpenAI Codex with Toast 1 exposed as a search sub-agent (low and high reasoning effort). Databricks Genie and model-provider harness results are as reported in the Databricks release; cost is per rollout in USD at list prices.