Mixedbread

Multipart Upload

Automatically upload large files in parallel chunks using the SDK's built-in multipart upload support for upload and uploadAndPoll.

Both upload and uploadAndPoll / upload_and_poll automatically use multipart upload for large files. The file is split into chunks and uploaded in parallel, improving reliability and speed. No extra configuration is needed — it just works.

Default Behavior

Files over 100MB automatically use multipart upload. The existing upload calls work without any changes:

Multipart Upload Default Behavior
file = mxbai.stores.files.upload(
  store_identifier="my-knowledge-base",
  file=Path("./large-document.pdf"),
)

Custom Threshold

Lower the threshold to trigger multipart upload for smaller files:

Multipart Upload with Custom Threshold
from mixedbread.lib.multipart_upload import MultipartUploadOptions

file = mxbai.stores.files.upload(
  store_identifier="my-knowledge-base",
  file=Path("./document.pdf"),
  multipart_upload=MultipartUploadOptions(
      threshold=50 * 1024 * 1024,  # 50MB instead of default 100MB
  ),
)

Custom Concurrency

Control how many parts upload in parallel. The default is 5 concurrent uploads:

Multipart Upload with Custom Concurrency
from mixedbread.lib.multipart_upload import MultipartUploadOptions

file = mxbai.stores.files.upload(
  store_identifier="my-knowledge-base",
  file=Path("./large-document.pdf"),
  multipart_upload=MultipartUploadOptions(
      concurrency=10,
  ),
)

Custom Part Size

Control the size of each chunk. The default part size is 100MB. Every part except the last must be at least 5MB, and an upload has at most 10,000 parts, so the default covers files up to about 1TB:

Multipart Upload with Custom Part Size
from mixedbread.lib.multipart_upload import MultipartUploadOptions

file = mxbai.stores.files.upload(
  store_identifier="my-knowledge-base",
  file=Path("./large-document.pdf"),
  multipart_upload=MultipartUploadOptions(
      part_size=50 * 1024 * 1024,  # 50MB parts
  ),
)

Progress Tracking

Get notified after each part finishes uploading. Works with both upload and uploadAndPoll / upload_and_poll:

Progress Tracking with uploadAndPoll
from pathlib import Path
from mixedbread import Mixedbread
from mixedbread.lib.multipart_upload import MultipartUploadOptions

mxbai = Mixedbread(api_key="YOUR_API_KEY")


def on_part_upload(event):
    pct = round((event.uploaded_bytes / event.total_bytes) * 100)
    print(f"Part {event.part_number}/{event.total_parts} done — {pct}%")


file = mxbai.stores.files.upload_and_poll(
    store_identifier="my-knowledge-base",
    file=Path("./large-document.pdf"),
    multipart_upload=MultipartUploadOptions(
        on_part_upload=on_part_upload,
    ),
)

print(file)

The callback receives a PartUploadEvent with the following fields:

FieldDescription
partNumber / part_number1-based part number that completed
totalParts / total_partsTotal number of parts in this upload
partSize / part_sizeSize of this part in bytes
uploadedBytes / uploaded_bytesCumulative bytes uploaded so far
totalBytes / total_bytesTotal file size in bytes
Multipart upload planner: choose a file size, threshold, partSize and concurrency to see how upload and uploadAndPoll split a file. A file larger than threshold (default 100 MB) is uploaded in parts; a smaller one goes up in a single request and onPartUpload is never called. The number of parts is ceil(file size / partSize): with the default partSize of 100 MB, a 1 GB file becomes 11 parts, ten of 100 MB and a last one of 24 MB. Up to concurrency parts (default 5) upload at the same time, and each lane takes the next part as soon as its current one finishes, so the 11 parts run as 5, 5 and 1. After every part, onPartUpload receives a PartUploadEvent with partNumber, totalParts, partSize (bytes in that part), uploadedBytes (cumulative) and totalBytes. Python uses the same options in snake_case: part_size and on_part_upload.
File size
threshold
partSize
concurrency
Upload
Multipart (1 GB > 100 MB)
Parts
ceil(1 GB / 100 MB) = 11, last part 24 MB
At once
min(5, 11) = 5

Each row uploads one part at a time and takes the next part when its current one finishes.

0%
partNumber
–
totalParts
11
partSize
–
uploadedBytes
–
totalBytes
10737418241 GB
Simulated timing is illustrative: every part moves at the same speed. Names follow the TypeScript SDK; Python uses part_size, on_part_upload and snake_case event fields.

All Options Together

Combine threshold, part size, concurrency, and progress tracking:

All Multipart Upload Options
from pathlib import Path
from mixedbread import Mixedbread
from mixedbread.lib.multipart_upload import MultipartUploadOptions

mxbai = Mixedbread(api_key="YOUR_API_KEY")


def on_part_upload(event):
    pct = round((event.uploaded_bytes / event.total_bytes) * 100)
    print(f"Part {event.part_number}/{event.total_parts} — {pct}%")


file = mxbai.stores.files.upload(
    store_identifier="my-knowledge-base",
    file=Path("./large-document.pdf"),
    multipart_upload=MultipartUploadOptions(
        threshold=50 * 1024 * 1024,  # 50MB threshold
        part_size=25 * 1024 * 1024,  # 25MB parts
        concurrency=10,
        on_part_upload=on_part_upload,
    ),
)

print(file)

Options Reference

OptionDefaultDescription
threshold100MBMinimum file size to trigger multipart upload
partSize / part_size100MBSize of each upload chunk, at least 5MB
concurrency5Number of parts uploaded in parallel
onPartUpload / on_part_upload—Callback invoked after each part completes

On this page

Default BehaviorCustom ThresholdCustom ConcurrencyCustom Part SizeProgress TrackingAll Options TogetherOptions Reference