Mixedbread Quality Evals
How Mixedbread performs on public retrieval and agent benchmarks, measured against the strongest competing models.
Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.
How we measure
Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.
Toast 1, our search agent, against frontier models running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.
LongSeal
Open SealQA paperHead-to-head as a standalone search agents: Toast 1 vs. GPT-5.6, Claude, Kimi, Qwen, GLM and DeepSeek running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.
- 1OPUS 523s0.78
- 2TOAST 1 (RRF ×3)7s0.71
- 3QWEN69s0.69
- 4TOAST 17s0.68
- 5DEEPSEEK118s0.68
- 6SONNET 522s0.66
- 7GLM96s0.66
- 8KIMI K322s0.65
- 9GPT-5.6 SOL11s0.62
- 10GPT-5.6 LUNA9s0.54
- 11HAIKU 4.533s0.50
- 12GPT-5.6 TERRA8s0.50
NDCG@10 vs. p50 latency per query and cost per query · higher is betterTested August 2026SealQA paperIntroducing Toast 1
Dataset and methodology
LongSeal is the needle-in-a-haystack variant of SealQA: each question pairs one answer-bearing document with up to 50 hard negatives that look relevant but aren't, testing whether a system can surface the document that actually answers the question amid convincing distractors.
Every system runs its full agentic search loop over the same index and returns a ranking; we report NDCG@10 of that ranking, cost per query at list prices with prompt caching, and p50 latency per query. Each frontier model is swept across its reasoning-effort settings; the chart connects each model's Pareto-efficient settings. Toast 1 runs on Mixedbread Search in its default configuration and in a three-way reciprocal-rank-fusion (RRF ×3) configuration.