Mixedbread Quality Evals

How Mixedbread performs on public retrieval and agent benchmarks, measured against the strongest competing models.

How we measure

Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.

Toast 1, our search agent, against frontier models running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.

BrowseComp-Plus

Open BrowseComp-Plus paper

Head-to-head as a standalone search agent: Toast 1 vs. GPT-5.6, Claude, Kimi, Qwen, GLM and DeepSeek running the same search loop over the same 100k-document index. Higher NDCG@10 at lower cost and latency per query is better.

  1. 1GPT-5.6 SOL159s0.86
  2. 2TOAST 1 (RRF ×3)10s0.86
  3. 3TOAST 18s0.80
  4. 4OPUS 539s0.79
  5. 5GPT-5.6 TERRA120s0.78
  6. 6GPT-5.6 LUNA41s0.73
  7. 7KIMI K3150s0.72
  8. 8QWEN108s0.67
  9. 9GLM111s0.66
  10. 10DEEPSEEK74s0.65
  11. 11SONNET 552s0.62
  12. 12HAIKU 4.552s0.40

NDCG@10 vs. p50 latency per query and cost per query · higher is betterTested August 2026BrowseComp-Plus paperIntroducing Toast 1

Dataset and methodology

BrowseComp-Plus is a deep-research eval with multi-hop questions over ~100k web documents. Here the search agent itself is evaluated: given the question, it must return the ranked evidence documents.

Every system runs its full agentic search loop over the same index and returns a ranking; we report NDCG@10 of that ranking, cost per query at list prices with prompt caching, and p50 latency per query. Each frontier model is swept across its reasoning-effort settings; the chart connects each model's Pareto-efficient settings. Toast 1 runs on Mixedbread Search in its default configuration and in a three-way reciprocal-rank-fusion (RRF ×3) configuration.