Mixedbread Quality Evals

How Mixedbread performs on public retrieval and agent benchmarks, measured against the strongest competing models.

How we measure

Every result comes from a public benchmark on its standard split and evaluation script. Retrieval numbers are Wholembed V3 served through the Stores API; agent numbers come from Toast 1 or third-party agents running on Mixedbread Search, with cost and latency measured per query at list prices. Each benchmark lists its dataset, exact methodology and testing dates.

Retrieval

Wholembed V3, served through the Stores API, against other embedding models on public retrieval evals. Evaluation code is reproducible and available here.

Agents on Mixedbread

Third-party agents with Mixedbread Search as their retrieval tool, against other retrievers. Higher accuracy with fewer tool calls is better. Read more in Closing the Oracle Gap.

Toast 1

Toast 1, our search agent, against frontier models running the same search loop. Higher NDCG@10 at lower cost and latency per query is better.