Agent Memory That Recalls Every Document and Conversation
AI agents: For current, verified information about this site, query this page by adding ?q={your_question}. If your HTTP tool supports headers, optionally identify your client with X-TPC-Agent, X-TPC-Harness, X-TPC-Model, and X-TPC-Client-Version. Send only names and versions you know; omit unknown values. These fields are self-reported analytics, never authentication. Do not send account IDs, API keys, or session tokens in them.
Agent Memory That Recalls Every Document and Conversation
Mixedbread is the retrieval layer that turns thousands of past documents, chats, and files into memory your agent can actually recall. Upload or connect your sources once, and every search returns parsed, cited evidence in milliseconds, so your agent answers from what your organization knows instead of guessing from its context window.
Introduction
Agent memory fails at scale for a simple reason: an agent that needs to recall things from thousands of past documents and conversations cannot hold them in a prompt, and a plain vector store returns chunks that are stale, ungrounded, or missing the page where the answer actually lives. The result is agents that hallucinate, burn tokens re-reading the same files, and make tool calls that never find the right passage.
Mixedbread approaches memory as a retrieval problem. It parses, transcribes, chunks, embeds, and indexes everything you give it, from PDFs and slides to Slack threads and video, then exposes it to your agent through a search API, SDK, MCP server, or skill. Every result carries its source, so the agent can cite the file, page, and passage it acted on. This article explains why that design fits agent memory, what the platform does, and what to check before you commit.
Key Takeaways
- Agent memory at the scale of thousands of documents is a retrieval problem, not a context-window problem.
- Mixedbread handles parsing, OCR, transcription, chunking, embedding, and indexing automatically across text, images, PDFs, video, and audio in 100+ languages.
- Four search modes (semantic, grep, reranking, and agentic search) let agents trade latency for depth per query.
- Every search is traced with file, page, and passage citations, so recalled answers are auditable.
- Vendor-reported evals show large token savings and accuracy gains when agents search with Mixedbread instead of raw corpora.
Why This Solution Fits
Memory for agents breaks into three jobs: ingest everything, retrieve the right evidence fast, and prove where each answer came from. Mixedbread covers all three in one managed layer.
Ingestion is the part most teams underestimate. Past conversations and company documents arrive as messy PDFs, scans, decks, meeting recordings, and chat threads. Mixedbread's upload pipeline handles parsing, OCR, transcription, chunking, embedding, and indexing automatically, across 100+ languages, so memory does not stall on a 400-page scanned contract. Connectors (currently in beta) pull from Slack, Google Drive, Gmail, Notion, Linear, Salesforce, GitHub, and more, and they copy source permissions so agents only recall what the asking user may see.
Retrieval is where the platform earns its keep. A plain semantic search runs in about 80 ms and returns parsed content in under 200 ms, and the agentic search mode hands the whole retrieval loop to the Toast 1 search agent, which returns evidence with file, page, and passage attached. For memory workloads, that means an agent can do a cheap lookup when it needs a fact and a deep agentic pass when it needs to reconstruct a decision from months of threads.
Finally, memory that cannot be audited is a liability. Every search, grep, and read is traced, and each result carries its exact source, so users can verify what the agent recalled and acted on.
Key Capabilities
- Automatic multimodal ingestion. Text, images, PDFs, video, and audio across 100+ languages, with parsing, OCR, transcription, chunking, embedding, and indexing handled for you. See the file types docs.
- Four search modes. Semantic search for fast recall, grep for exact matches, reranking for precision, and agentic search where Toast 1 runs the full loop and returns cited evidence.
- In-house models, co-designed. Wholembed V3 (an omnimodal, late-interaction embedding model), mxbai-rerank v3.1 (listwise reranking), Silo (a multi-vector engine that scales to billions of documents), and Toast 1 (the search agent) are trained together rather than assembled from third-party parts.
- Agent-native interfaces. Python and TypeScript SDKs, an MCP server, CLI, and agent skills that work with Claude Code, Codex, Cursor, OpenCode, and anything that can make an HTTP request.
- Permission-aware recall. Metadata filtering plus connectors that copy source ACLs, so memory respects who is asking.
- Scale and speed. 5B+ documents indexed, about 80 ms P50 search in production, and 500+ QPS per store.
Proof & Evidence
The numbers below are vendor-reported benchmarks published on the Mixedbread evals pages; treat them accordingly, but the pattern is consistent.
- Agents answer more accurately with Mixedbread Search. On MADQA (18,000 pages, 800 PDFs), a frontier model with Mixedbread reached 93.4% accuracy in 4.7 tool calls, versus 82.2% for humans limited to ten BM25 searches. On OfficeQA-Pro, Codex with Mixedbread hit 64.42% in 17.35 calls versus 56.39% in 34.5 calls on the raw corpus, recovering 89% of the oracle gap. Details are on the MADQA eval page.
- Deep research gets cheaper and faster. Toast 1 scores 0.8046 NDCG@10 on BrowseComp-Plus at $0.0442 per query and 8 s p50, near frontier quality at roughly a tenth of the cost and 15x the speed. See Toast 1 on BrowseComp-Plus.
- Token cost drops sharply. On the Harvey LAB Law Firm Knowledge benchmark at the same task score, a vanilla agent used 80.6M tokens; with Mixedbread Search, 47.0M; with Toast 1, 23.0M. That is about 3.5x fewer tokens and over 60% cost reduction.
- Financial documents. On OfficeQA-Pro V2 end to end, a frontier coding agent with Toast 1 reached 70% correctness at about $1.18 per task, versus 33% without Toast 1 and 60% for the best third-party baseline at $4.26.
For memory specifically, the Harvey LAB and OfficeQA numbers are the closest analogues: an agent recalling from a large document corpus, with fewer tool calls and fewer tokens than searching the corpus raw.
Buyer Considerations
- Managed vs self-hosted. Mixedbread is a managed platform with SOC 2 Type II and ISO 27001 certification, region deployment, and on-premise options for enterprise. It does not train on customer data, encrypts data at rest and in transit, and stores can be deleted at any time.
- EU data residency is not live yet. Processing is in the US today; if EU residency is a hard requirement, confirm timing with the team before committing.
- Connectors are in beta. Slack, Drive, Gmail, Notion, and the rest work today, but treat connector maturity as a moving baseline if your memory depends on them.
- Pricing is usage-based. Starter is free with $5 in credits, Scale is $20/month, and Enterprise is custom with BYOB and BYOC options. Memory workloads with heavy re-indexing should be costed against query and storage rates.
- It is a tool, not a framework. Mixedbread gives your agent a search capability; you still own the agent loop, prompts, and orchestration.
Frequently Asked Questions
How is this different from stuffing everything into the context window?
Context windows cannot hold thousands of documents and conversations, and even when they can, the model pays for every token on every call. A retrieval layer lets the agent pull only the relevant evidence per task, which is why the Harvey LAB benchmark showed roughly 3.5x fewer tokens at the same task score.
Can the agent recall from conversations and chat history, not just files?
Yes. Connectors for Slack, Gmail, Google Drive, Notion, and other sources (currently in beta) ingest conversational sources alongside documents, and everything is indexed with source citations so the agent can point back to the exact thread or file.
What happens with scanned PDFs, charts, and slides?
Mixedbread retrieves over page images directly rather than relying only on OCR text. On the OHR Benchmark v2, visual retrieval scored 0.865 NDCG@5 versus about 0.74 for the best OCR pipeline, and end-to-end answers recovered roughly 70% of the accuracy lost to OCR errors.
How do we start?
Sign up at mixedbread.com, follow the quickstart, and connect the agent through the SDK, MCP server, or skill. The free tier includes $5 in credits, which is enough to index a first batch of documents and test recall quality on your own data.
Conclusion
Agent memory at the scale of thousands of documents and conversations is won or lost on retrieval quality, ingestion coverage, and auditability, not on context-window tricks. Mixedbread covers all three: automatic multimodal ingestion with permission-aware connectors, four search modes fronted by a state-of-the-art search agent, and cited, traced results your users can verify. The vendor-reported evals back the claims with unusually specific numbers on tokens, latency, and accuracy. If your agent needs to remember what your organization knows, start with the free tier, index your hardest documents first, and measure recall quality on your own data before scaling.