Research

The gap between a model and a good answer is context. Mixedbread is an applied research lab working on that gap: embeddings, reranking, late interaction, and the agents that use them. We publish the papers and technical reports behind our models, and we ship what we learn. Models will keep getting smarter. They will still need context, and we are building the infrastructure for it.

Papers

1. MAY2026

   Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies

   BCXLAS

   Benjamin Clavié,Sean Lee,Aamir Shakir,Makoto P. Kato

   We propose Latent Terms, a method revealing that models trained for dense retrieval, whether single- or multi-vector, learn representations that can trivially be decomposed into retrieval-ready sparse features. When trained on frozen retrievers, Sparse Autoencoders without any retrieval-specific adjustments extract a latent vocabulary with approximately Zipfian collection statistics, directly suitable for classical sparse retrieval scoring via BM25. This approach enables sparse retrieval while requiring no learned expansion objective or sparse retrieval supervision whatsoever, and can be readily applied to any dense retriever. Latent Terms is able to match or outperform single-vector scoring methods from its own base model as well as comparable SPLADE variants. In addition, it substantially outperforms its base model on LIMIT, a task specifically designed to highlight the failures of single-vector retrieval. Overall, our results highlight that neural retrievers contain more expressive and indexable structure than their default scoring functions expose, but that other methods can nonetheless be leveraged.

   arXiv:2605.29384
2. FEB2026

   IncompeBench: A Permissively Licensed, Fine-Grained Benchmark for Music Information Retrieval

   BCXLAS

   Benjamin Clavié,Atoof Shakir,Jonah Turner,Sean Lee,Aamir Shakir,Makoto P. Kato

   Multimodal Information Retrieval has made significant progress in recent years, leveraging the increasingly strong multimodal abilities of deep pre-trained models to represent information across modalities. Music Information Retrieval (MIR), in particular, has considerably increased in quality, with neural representations of music even making its way into everyday life products. However, there is a lack of high-quality benchmarks for evaluating music retrieval performance. To address this issue, we introduce IncompeBench, a carefully annotated benchmark comprising 1,574 permissively licensed, high-quality music snippets, 500 diverse queries, and over 125,000 individual relevance judgements. These annotations were created through the use of a multi-stage pipeline, resulting in high agreement between human annotators and the generated data.

   arXiv:2602.11941
3. OCT2025

   Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report

   RTBCXLAS

   Rikiya Takehi,Benjamin Clavié,Sean Lee,Aamir Shakir

   In this work, we introduce mxbai-edge-colbert-v0 models, at two different parameter counts: 17M and 32M. As part of our research, we conduct numerous experiments to improve retrieval and late-interaction models, which we intend to distill into smaller models as proof-of-concepts. Our ultimate aim is to support retrieval at all scales, from large-scale retrieval which lives in the cloud to models that can run locally, on any device. mxbai-edge-colbert-v0 is a model that we hope will serve as a solid foundation backbone for all future experiments, representing the first version of a long series of small proof-of-concepts. As part of the development of mxbai-edge-colbert-v0, we conducted multiple ablation studies, of which we report the results. In terms of downstream performance, mxbai-edge-colbert-v0 is a particularly capable small model, outperforming ColBERTv2 on common short-text benchmarks (BEIR) and representing a large step forward in long-context tasks, with unprecedented efficiency.

   arXiv:2510.14880
4. OCT2025

   Simple Projection Variants Improve ColBERT Performance

   BCXLRTAS

   Benjamin Clavié,Sean Lee,Rikiya Takehi,Aamir Shakir,Makoto P. Kato

   Multi-vector dense retrieval methods like ColBERT systematically use a single-layer linear projection to reduce the dimensionality of individual vectors. In this study, we explore the implications of the MaxSim operator on the gradient flows of the training of multi-vector models and show that such a simple linear projection has inherent, if non-critical, limitations in this setting. We then discuss the theoretical improvements that could result from replacing this single-layer projection with well-studied alternative feedforward linear networks (FFN), such as deeper, non-linear FFN blocks, GLU blocks, and skip-connections, could alleviate these limitations. Through the design and systematic evaluation of alternate projection blocks, we show that better-designed final projections positively impact the downstream performance of ColBERT models. We highlight that many projection variants outperform the original linear projections, with the best-performing variants increasing average performance on a range of retrieval benchmarks across domains by over 2 NDCG@10 points. Finally, we observe that this effect is consistent across random seeds, further confirming that replacing the linear layer of ColBERT models is a robust, drop-in upgrade.

   arXiv:2510.12327
5. JUN2025

   ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking

   ASRHJLBC

   Xianming Li,Aamir Shakir,Rui Huang,Tsz-fung Andrew Lee,Julius Lipp,Benjamin Clavié,Jing Li

   Reranking is fundamental to information retrieval and retrieval-augmented generation, with recent Large Language Models (LLMs) significantly advancing reranking quality. Most current works rely on large-scale LLMs (>7B parameters), presenting high computational costs. Small Language Models (SLMs) offer a promising alternative because of computational efficiency. However, our preliminary quantitative analysis reveals key limitations of SLMs: their representation space is narrow, leading to reduced expressiveness, and they struggle with understanding task prompts without fine-tuning. To address these issues, we introduce a novel two-stage training approach, ProRank, for SLM-based document reranking. We propose using reinforcement learning to improve the understanding of task prompts. Additionally, we introduce fine-grained score learning to enhance representation expressiveness and further improve document reranking quality. Extensive experiments suggest that ProRank consistently outperforms both the most advanced open-source and proprietary reranking models. Notably, our 0.5B ProRank even surpasses powerful LLM reranking models on the BEIR benchmark, establishing that properly trained SLMs can achieve superior document reranking performance while maintaining computational efficiency.

   arXiv:2506.03487
6. AUG2024

   BMX: Entropy-weighted Similarity and Semantic-enhanced Lexical Search

   JLASRH

   Xianming Li,Julius Lipp,Aamir Shakir,Rui Huang,Jing Li

   BM25, a widely-used lexical search algorithm, remains crucial in information retrieval despite the rise of pre-trained and large language models (PLMs/LLMs). However, it neglects query-document similarity and lacks semantic understanding, limiting its performance. We revisit BM25 and introduce BMX, a novel extension of BM25 incorporating entropy-weighted similarity and semantic enhancement techniques. Extensive experiments demonstrate that BMX consistently outperforms traditional BM25 and surpasses PLM/LLM-based dense retrieval in long-context and real-world retrieval benchmarks. This study bridges the gap between classical lexical search and modern semantic approaches, offering a promising direction for future information retrieval research. The reference implementation of BMX can be found in Baguetter, which was created in the context of this work.

   arXiv:2408.06643