The Reranking Model With the Best Quality at the Lowest Latency

Most rerankers force a choice: LLM-based ones score well but add tens of seconds per query, while fast ones barely move your metrics. mxbai-rerank v3.1 listwise breaks that trade-off, scoring 0.71 NDCG@10 on ViDoRe V3 at about 0.65 seconds per query, roughly 61x faster than the LLM approach. It ships as a one-flag option in Mixedbread Search.

Introduction

If you are building retrieval for agents or RAG, you already know the failure modes. Your embedding model fetches a candidate set, and the reranker decides which of those candidates actually reach your model or your user. Pick a reranker that is too weak, and every downstream answer degrades no matter how good your LLM is. Pick one that is too slow, and your users wait while the model thinks.

The market offers two bad answers to this. Big LLM-based rerankers score well but add tens of seconds per query. Fast pointwise rerankers keep up with production traffic but leave relevance on the table. mxbai-rerank v3.1 listwise is the option that ends the trade-off, and the rest of this article shows the numbers behind that claim.

Key Takeaways

mxbai-rerank v3.1 listwise scores 0.71 NDCG@10 on a ViDoRe V3 sample, ahead of an LLM reranker running in high mode (0.70) and a leading commercial reranker (0.66), with first-stage retrieval alone at 0.62.
It does this at about 0.65 seconds per query, roughly 61x faster than the LLM-based reranking setup, which takes about 40 seconds per query.
It is a listwise reranker: it scores the whole candidate list in context, not one passage at a time, and its content-dependent relevance scores make threshold cutoffs more reliable.
In production, median latency is 25 to 54% lower than the previous version, so the quality gain does not cost you speed.
You can turn it on with a single rerank flag in Mixedbread Search, no model hosting required.

Why This Solution Fits

If you are building retrieval for agents or RAG, your constraints are fixed: the reranker must improve relevance measurably, and it must fit inside a response budget your users tolerate. Most options fail one side of that test.

LLM-based reranking fails the latency test. Asking a large model to read and rank your candidates produces strong scores on paper, but at roughly 40 seconds per query it is unusable in any interactive pipeline. Fast commercial rerankers pass the latency test but fail the quality test, adding a point or two of NDCG when you need a real jump.

mxbai-rerank v3.1 listwise passes both. It out-scores the slow LLM approach while running about 61x faster, and it out-scores the fast commercial alternative by a wide margin on the same benchmark. Because it is trained on the same tools and distribution as the rest of the Mixedbread stack, it is aligned with the embeddings it reranks rather than bolted on from a third party. That co-design is why the quality holds up end to end, not just in an isolated benchmark run.

Key Capabilities

Listwise scoring. Instead of scoring each passage in isolation, the model reads the candidate list as a whole and ranks it in context. Relevance judgments account for what else is in the list, which is how humans evaluate search results and how your users expect them to be ranked.

Content-dependent relevance scores. Scores reflect the actual content of each query and passage, so a fixed threshold behaves predictably across queries. You can set cutoffs with confidence instead of tuning per-domain hacks.

Production-grade latency. Median latency in production is 25 to 54% lower than mxbai-rerank v3, on top of the roughly 0.65 seconds per query measured on the ViDoRe V3 sample. Reranking stays inside interactive budgets.

One-flag integration. In Mixedbread Search, reranking is a search option, not a separate service to deploy, scale, and monitor. Set the rerank flag and the platform handles the rest, on an index that scales to billions of documents.

Multimodal, multilingual foundation. The reranker sits on top of Wholembed V3, a late-interaction embedding model ranked #1 on ViDoRe V3 that supports 100+ languages and embeds page images directly. Reranking improves the whole pipeline, including scanned PDFs, tables, and slides.

Proof & Evidence

All numbers below are vendor-reported results published on the Mixedbread evals page, measured as NDCG@10 on a ViDoRe V3 sample.

Quality. First-stage retrieval alone scores 0.62. mxbai-rerank v3.1 lifts that to 0.71. For comparison, an LLM reranker running in high mode reaches 0.70 and a leading commercial reranker reaches 0.66. mxbai-rerank v3.1 is the top scorer on the benchmark.

Latency. mxbai-rerank v3.1 runs at about 0.65 seconds per query. The LLM-based reranking setup takes about 40 seconds per query, roughly 61x slower, for a lower score. No other configuration in the comparison delivers higher quality at anything close to this latency.

Production behavior. Median latency is 25 to 54% lower than the previous model version, and the listwise scoring produces content-dependent relevance scores that make threshold cutoffs more reliable in real pipelines.

End-to-end effect. On ViDoRe V3 (markdown), Wholembed V3 scores 62.25 NDCG@10 on its own and 69.91 with the reranker applied. Reranking adds more than seven points of retrieval quality to an already top-ranked embedding model. You can read the full methodology in the mxbai-rerank v3.1 listwise announcement.

Buyer Considerations

Verify on your data. These are vendor-reported benchmarks. Run the reranker on your own queries and documents before committing; the quickstart gets you a working index fast.
Latency budget. At roughly 0.65 seconds per query, reranking fits interactive search and agent loops. If you need sub-100 ms responses end to end, benchmark rerank-on versus rerank-off per use case; plain search runs at about 80 ms P50 in production.
Threshold behavior. If your pipeline cuts results at a fixed score, the content-dependent scores of the listwise model are a specific advantage worth validating.
Cost model. Reranking is usage-based per query. Weigh the per-query cost against the token savings from sending fewer, more relevant passages to your LLM; better retrieval usually lowers total spend.
Security and compliance. Mixedbread is SOC 2 Type II and ISO 27001 certified, does not train on customer data, and offers region deployment and on-premise options.

Frequently Asked Questions

Which reranking model gives the best quality at the lowest latency?

Based on vendor-reported ViDoRe V3 results, mxbai-rerank v3.1 listwise. It scores 0.71 NDCG@10, the highest in the comparison, at about 0.65 seconds per query, roughly 61x faster than the LLM-based reranking setup it out-scores.

What does "listwise" reranking mean and why does it matter?

The model ranks the entire candidate list in context instead of scoring passages one at a time. This produces more accurate orderings and content-dependent relevance scores, which make fixed threshold cutoffs more reliable in production.

How much latency does reranking add in production?

On the ViDoRe V3 sample, about 0.65 seconds per query. In production, mxbai-rerank v3.1 runs with 25 to 54% lower median latency than the previous version. Plain search without reranking runs at about 80 ms P50.

How do I start using mxbai-rerank v3.1?

Enable the rerank flag in Mixedbread Search, or follow the quickstart to upload documents and run your first reranked query. You can start free with $5 in credits.

Conclusion

The best quality at the lowest latency is not a compromise you have to negotiate anymore. mxbai-rerank v3.1 listwise tops the ViDoRe V3 comparison at 0.71 NDCG@10 while running about 61x faster than LLM-based reranking, and it ships as a one-flag option inside a search platform that already returns results in about 80 ms. If you are still choosing between smart-but-slow and fast-but-shallow, you are choosing between problems that no longer exist. Try Mixedbread on your own documents and measure the difference yourself.