Building a Robust Ingestion System for Any File of Any Size

Mixedbread lets users upload arbitrary files. No matter the modality, language, size or shape, we ingest, index and make it searchable.

In our early days, we had tested our ingestion pipeline with 1MB files and shipped it as is, expecting it to cover the needs of most of our users. Very shortly after, some users started uploading large knowledge bases, with text files exceeding 1GB being a lot more common than we had expected. At the time, that crashed our worker on the spot. Later on, as multimodal use cases became more and more common, users started uploading hundreds of gigabytes of video files and expected them to just work. To ensure that we could process arbitrarily large files as our users' needs grow, we had to rebuild our ingestion logic from the ground up.

Take a brief look at your ~/Downloads folder: it probably contains a mess of file types and sizes. In the real world, most systems avoid this mess by only accepting carefully constructed inputs with strict limits and formats. But every additional restriction results in frustration for the user. Remember how big the demand for "Chat with your PDF" was back in 2023, to the point that hundreds of apps appeared to serve this need? Every strict file limit is an unmet user need.

This realization has resulted in considerable complexity for teams: in both agentic and retrieval systems, building scalable ingestion pipelines automatically becomes a major part of the work. This leads to many, many teams spending a considerable chunk of their time building these systems through trial and error.

At Mixedbread, our aim is to solve this problem, and to let you focus on building your application rather than fighting with its building blocks. To do so, we ensured that our retrieval models are co-designed with our processing pipelines. But that wasn't enough. In order to provide the best possible quality and user experience, we not only needed a strong, co-designed pipeline for common inputs, we also had to allow our users to upload any file to Mixedbread, without requiring pre-processing on their end.

Don't Trust FilesLink to section

While designing and building this, we encountered three major challenges.

• Files are big: A multi-hour video recording is hundreds of gigabytes. In a multi-tenant system multiple users might upload very large files at the same time. Loading everything upfront into memory is prohibitive. Scaling the machine to the file does not work either: machines with enough memory for the worst case are expensive, slow to provision, and sit idle the rest of the time. And there is always a bigger file.

• Small files blow up: File size is a terrible indicator for processing cost. A 4MB PDF can contain 2,000 pages, and we have to render every page with sufficient quality for our multimodal models to represent every small nuance. This rendering can result in transitional files that are orders of magnitude larger than the heavily compressed inputs. But the opposite is not uncommon either: a 50MB PDF might simply be a dozen photographed pages without much content. Any system budgeting resources by input size will find itself struggling to handle both of these extremes, and eventually stall the whole pipeline.

• Files are unsanitized: Word and PowerPoint files reference fonts that are not installed on the machine rendering them. Slides contain embedded screenshots of other documents. Real-world office files are best treated as untrusted programs that happen to produce pages.

Ultimately, these three problems largely obscure a fourth, overarching one: it is necessary to be able to estimate the processing cost of a file at upload time, before any parsing steps. Otherwise, it is impossible to reliably check quotas and provide users with predictable billing. We'll talk more about that in a minute.

The Problem with Fixed-Size SplittingLink to section

The classic answer to "file too big" is to cut it into fixed-size byte ranges and process them independently. For search quality, this is a bad idea. A byte boundary might land in the middle of a sentence, a table, a video scene, or a compressed frame group. Our retrieval quality depends on chunks being meaningful semantic units, which is why our parsers split text at logical boundaries, audio at low-energy regions, and video at scene changes.

So we had two requirements that pull in opposite directions: process files in bounded pieces, but choose chunk boundaries semantically. The solution was to separate those two concerns into two components that operate at different granularities.

Slicer and ParserLink to section

The slicer concerns which bounded region of the file should be processed next. The parser takes care of how we turn that region into high-quality, searchable chunks. The slicer cuts coarsely and cheaply, the parser cuts finely and semantically within each slice.

Processing a file works like this:

1. Probe: On the first invocation, we determine the file's total extent. Each file type has a different unit to express that extent: for a PDF, it is simply the page count. For videos, we run ffprobe against a presigned URL and extract the duration from the container metadata, without needing to download the file.
2. Slice: The slicer's role is simple: it predicts the next bounded range and prepares it for the next steps, so that each slice can be processed sequentially. Again, the bounded range varies by file type: for PDFs, the first slice is pages 0 to 32; for videos, it is seconds 0 to 300.
3. Parse: The parser processes only that range. Within a single slice, it is tractable to perform expensive, semantically relevant work: rendering pages, detecting scene boundaries, transcribing audio, splitting text at logical breaks, and so on. The parser works on a continuously streaming basis, never accumulating individual chunks in memory.
4. Baton-passing: After the parser is done, the worker checks whether the file has content left. If it does, it queues up the next slice with a small message, containing information about where it starts, its total extent, any probed metadata, and running totals used for both chunk counting and billing. This is picked up by a fresh worker, which continues exactly where the previous one stopped.
5. Complete: Once a worker identifies that there is no more work to be done, the final slice merges all running totals and marks the file as completed.

The important property of this loop is predictable work delimitation: every invocation does a bounded amount of work, unrelated to how large a given file is. The slicer's state is always a few hundred bytes, and a worker never holds more than a single slice at any given time. In practice, this means that the only difference between a 3-hour video and a 30-second clip is the number of individual invocations. They both share identical code paths and the same per-worker workload.

Since queue messages fully describe each individual slice, every invocation is retryable and idempotent, which makes the pipeline robust to individual failures. When a worker dies mid-slice, whether from a timeout, an OOM, or a hiccup in an external service, its message re-enters the queue and a fresh worker redoes just that slice. Any single failure costs one slice retry, and leaves the rest of the processing entirely unharmed.

Choosing a Slice Unit and SizeLink to section

The probe and the slicer both lean on the idea that every file type has a unit that expresses its extent. Pages for a PDF, seconds for a video. That unit is also what the slicer counts in when it decides how much of the file the next worker gets, so each file type slices in its own natural unit:

The slice sizes may look arbitrary, but they are in fact chosen so that the worst realistic slice fits comfortably into a fixed worker size and timeout. The budget math for the two heaviest types:

PDF slice    32 pages × ~26 MB per page rendered at 300 DPI   ≈ 0.8 GB
video slice  300 s fetched over HTTP at ~1 MB/s               ≈ 0.3 GB

Both of these slices leave ample room on a 2GB worker, and even in particularly challenging cases, any given 5-minute video slice can be fetched, scene-detected, and re-encoded safely within a 15-minute window. Since each file is processed as a series of slices, ensuring that slice sizes are safe is enough to make the whole file safe to process, as the workers only ever approach the file as a sequence of slices.

For a concrete example, take a 3-hour lecture recording. First, a single ffprobe call against a presigned URL reads the duration, 10,800 seconds, from its container metadata. This results in roughly 36 slices of about 5 minutes each, leading to 36 worker invocations. Each of them fetches only its own assigned 5-minute window, then performs the steps detailed above: split it at scene boundaries into chunks of 8 to 25 seconds, transcribe them, and append them to the store. Around 700 chunks later, the final worker notices that the file extent is exhausted, merges the running totals, and marks the file as completed.

Preserving Chunks Across Edge CasesLink to section

However, one issue remains with a pure slice-based approached: slicing, by nature, occurs before any parsing is performed, meaning that we do not know where natural semantic boundaries within the input document will fall. Each slice has a fixed, rule-based end, and real chunks are likely to be split between two slices. This is one of the reasons why our slices are dimensioned with headroom: in all cases where our parser detects a chunk has not reached its natural end by the end of the slice, the record we enqueue for the next slice marks its start as the end of the last fully-formed chunk, rather than the end of the full slice. This leads to Slice n+1 beginning exactly on a semantic boundary, and ensuring that the straddling chunk is processed in full.

This is also why our system is built to be inherently sequential: slice n+1 cannot have its starting point fully defined until slice n has decided on its semantically logical endpoint. We deliberately trade away within-file parallelism in exchange for greater meaning preservation, as it is a minor, one-time cost that preserves long-term quality.

Keeping Memory Bounded Inside a SliceLink to section

Slicing bounds how much of the file a worker sees, but a single slice can still cause issues. A few guards inside the parsers help prevent any major issue:

• Streaming instead of downloading. The parsers for heavier file types, video and audio, are specifically designed so that the full source file is never downloaded. ffmpeg and ffprobe are used to read presigned URLs directly over HTTP range requests, so that extracting minutes 55 to 60 of a huge video only needs to fetch the exact bytes for this timestamp.
• Render limits per page. PDFs can contain and declare enormous page geometries. The DPI per page is automatically adjusted in order to preserve as much information as possible while ensuring no single render can exceed a fixed pixel budget. This leads to malformed page rendering at a reasonable, smaller DPI, leaving the worker safe.
• Lazy chunk iteration. All parsers yield individual chunks as async iterators, and they are written out one at a time. The full chunk list for a slice never occupies memory.

These considerations mean that bounded, predictable memory usage is a property architecture. This allows us to have our ingestion worker be ordinary 2GB machines (8GB for the high-quality parsing path), which can be scaled horizontally on demand with virtually no overhead.

Estimating Cost before IngestionLink to section

When a file is attached to a store, we first check quotas before beginning ingestion. At that point, we know virtually nothing about the file, so the quota estimation is done in tiers thanks to cheap, easily accessible information:

Each modality has its own cheap indicator: for small PDFs, we can read the page count. For audio and video, duration is a cheap proxy, requiring just a single ffprobe call. For everything else, whether it be raw text or office documents, estimates from bytes are usually very reliable. For example, Office documents surprisingly, yet consistently, average at roughly 50KB per page, and being off by a few pages is acceptable as the exact count will be reconciled later during ingestion.

What's NextLink to section

Currently, slices for any given input file are processed serially in order to allow for the semantic boundary handoff described above. While this is fine for the vast majority of uses, this does result in very large files needing long processing times, as wall-clock time grows linearly with file size. And as we also mentioned earlier, we foresee that the size of input files will continue to increase as model capability grows and new uses get unlocked. For v2 of our ingestion pipeline, we have already identified ways to ensure that independent slices can fan out to workers concurrently, with their semantic boundaries being pre-estimated and a merging process that ensures all semantic information is preserved despite parallelism.

Try it NowLink to section

Mixedbread is the retrieval infrastructure for AI agents. Retrieve evidence across text, PDFs, tables, images, and videos with state-of-the-art accuracy.

from mixedbread import Mixedbread
from pathlib import Path

client = Mixedbread()

client.stores.create(name="compliance-docs")

client.stores.files.upload_and_poll(
    store_identifier="compliance-docs",
    file=Path("soc2-report.pdf")
)

client.stores.search(
    store_identifiers=["compliance-docs"],
    query="is there a secure SDLC?"
)

If you have files that other systems refuse to touch, we would genuinely like to see them. Upload them on our platform (here are all the supported file types), and if you manage to crash an ingestion worker, tell us on Slack. It has happened before, and every time it made the pipeline better. And if you are interested in making the most obscure files work regardless of size, consider applying here.