Build Your Own Harness

Toast is a deep search and lookup agent, trained to submit ranked evidence. You can use Toast with our internal Toast-1 harness directly via our Search API with agentic search feature.

Like most agents, Toast is only as good as its tools. It performs best with a late-interaction style semantic search, but has no issues working with your other search backends. The search agent adapts well to new tools and custom harnesses: file search tools, tools to access domain knowledge, or custom answer tools. This guide collects recommendations and guidance for defining custom harnesses with our Completions API so you can make best use of Toast-1's strengths for your use case.

Toast-1 is trained for deep evidence gathering. It prefers to fan out searches via parallel search calls, resulting in high exploration for minimal latency. It's the perfect multilingual exploration agent for knowledge work: a substitute for luna, terra, or haiku/sonnet as a look-up subagent. If these characteristics match your use case, follow this guide to create your own fast searcher.

Platform constraintsLink to section

• Toast has thinking disabled. It was trained without thinking and is only served without thinking to have state-of-the-art retrieval performance at minimum latency.
• Sampling parameters are configurable; we recommend temperature 0.7, top_p 0.95.
• Toast is served at seq_len = 131072: every input message needs to stay below 130k tokens.
• Generation is capped at 4096 tokens per completion.

Manage and compact your context against this limit (for example, use it for a Pi harness internal compaction). The Completions API does no context management for you.

In our internal Toast-1 harness, Toast uses pruning as its context management: a prune_context tool the agent calls to remove chunks it considers irrelevant. We instruct Toast to call this tool when it's approaching the context limit. Prefer prune-style removal of stale tool results over summarizing compaction; our inference and latency are optimized for this.

Do not use previous_completion_id after your context management has edited the history. It only works when the history is resent unchanged. After pruning, just resend the full edited messages list (see the loop example at the end of this guide).

Harness design & Completions API guidanceLink to section

The Completions API is an OpenAI Chat-Completions-compatible endpoint: provide your prompt, messages, tool definitions, and sampling params exactly the way you would with the standard OpenAI client. You send a messages list plus your tool schemas in tools, and Toast answers with either text or tool calls. The API is stateless: every request resends the full message history, and the server executes nothing for you. When Toast calls one of your tools, the completion ends with finish_reason: "tool_calls"; your harness executes the calls, appends one tool message per call, and sends the next request. That loop is your whole harness.

The following guidelines are design choices inspired by our internal Toast-1 harness. Treat them as recommendations, not requirements.

• Toast is trained to work within a bounded number of turns and a bounded number of parallel tool calls. Communicate those boundaries in the harness. Our hosted agentic search defaults to 4 turns with up to 8 parallel calls per turn, but this is a soft recommendation; Toast can easily go up to 8, 12, or more turns.
• For similar retrieval performance, enable parallel tool calls in your harness and make your tools safe to execute concurrently. Parallel tool latency is bounded by the slowest tool you include.
• Do not overflow the 130k token context limit when providing the full results of many parallel calls back to the agent. With up to 8 parallel calls per turn, uncapped tool returns fill the context quickly. If your tools return docs with many tokens, truncate them.
• Return short stable handles with your search tools' results, like a short chunk_id. Every result your search tools return needs a stable id, and any tool that takes other tools' outputs as inputs must accept exactly the ids you emitted. Tools that operate on retrieved results are easy to define this way.
• Deduplicate if you want: filtering already seen search results out of the tool call results leads to less context rot and higher recall.
• Give Toast a tool to end the searching and report its findings. Our internal Toast-1 harness uses a reporting tool that takes the chunk_ids your search tools emitted plus a relevance score per chunk and quick reasoning to answer the input query.
• Define your tools to return a dict, not prose text. Our Toast-1 harness tool results are JSON envelopes: {"query": ..., "results": [...], "candidate_count": N} with per-result entries like {"chunk_id": ..., "text": ..., "score": ..., "metadata": {...}}. Prefer returning a similar shape from your functions.
• Communicate tool schema errors back to Toast instead of raising them. If you provide the tool error in the tool response, Toast can retry the call in the next turn; we trained Toast to recover well from structured errors. Example: catch the exception and return {"error": "date must be ISO format, got '3/5/24'"}.

Phrasing of instructionsLink to section

Toast performs best writing human-style questions for a semantic search tool and writing regex for grep tools. Be precise in your custom tool definitions and give clear instructions on how to use the tool (see the example at the end of this guide).

When adding custom search tools, providing instructions on how to formulate queries helped a lot to boost retrieval performance. An example for a BM25 tool: "this BM25 tool is matching keywords, write only keyword-heavy queries".

Build your own harness with PythonLink to section

For a custom searcher harness in Python, write the tool descriptions directly in the docstrings and type hints of your tool functions. Docstrings and type hints are the interface Toast sees: the Python harness generates the JSON tool schema from them — docstring → tool description, Annotated strings → parameter descriptions.

• Toast follows tool descriptions as instructions. Say what the tool matches on, when to prefer it, and how to phrase input ("this tool matches keywords only. Send keyword-heavy queries, not questions").
• Keep tool signatures flat and JSON-native. Toast was trained on flat argument objects: str, float, bool, list[str], and one level of typed dicts.
• Prefer snake_case names. Mark what's essential as required and give defaults for the rest of your parameters.
• Use Literal/Enum for closed choices. Trained tools express modes as enums (filter_mode: Literal["all", "any"], direction: Literal["asc", "desc"]).

Example 1: a simple semantic search tool with just a query and a top_k parameter.

from typing import Annotated


def semantic_search(
    query: Annotated[str, "Natural-language query for a single search aspect; "
                          "avoid Boolean syntax, regex, and keyword dumps."],
    top_k: Annotated[int, "Number of chunks to return, max 20."] = 5,
) -> dict:
    """Execute a meaning-based semantic search query over the corpus and return the
    most relevant chunks. Use natural language; phrase queries as human-style
    questions. Do not use for keyword, regex, or literal-string matching.
    Returns up to top_k chunks with stable chunk_id handles."""
    hits = my_search_backend(query, top_k=top_k)  # your implementation
    return {
        "query": query,
        "candidate_count": len(hits),
        "results": [
            {"chunk_id": h.id,                   # short stable handle, e.g. "c12"
             "score": round(h.score, 4),
             "text": h.text,                     # clipped, not the whole document
             "metadata": h.metadata}
            for h in hits
        ],
    }

Example 2: a custom BM25 keyword-search tool.

from typing import Annotated, Literal


def bm25_search(
    query: Annotated[str, "Space-separated keywords, no natural-language questions, "
                          "no boolean operators. Example: 'jordan international goals caps'"],
    top_k: Annotated[int, "Number of chunks to return, max 20."] = 5,
    mode: Literal["chunks", "documents"] = "chunks",
) -> dict:
    """Keyword-based BM25 search over the corpus. This tool matches keywords only.
    Send keyword-heavy queries, not questions. Use for rare terms, names, codes,
    and exact vocabulary; Returns up to top_k chunks with stable chunk_id handles."""
    hits = bm25_index.search(query, k=top_k, mode=mode)  # your implementation
    return {
        "query": query,
        "candidate_count": len(hits),
        "results": [
            {"chunk_id": h.id,                   # short stable handle, e.g. "c12"
             "score": round(h.score, 4),
             "text": h.text,                     # clipped, not the whole document
             "metadata": h.metadata}
            for h in hits
        ],
    }

Note how the description tells Toast what the tool is for, how to phrase input, when not to use it (and which tool to use instead), and how many results to expect — and the return value is a JSON envelope with short stable handles and a score.

Build your own harness with PiLink to section

If you use Pi as your agent framework, the harness loop already exists: Pi runs the tool-call loop, executes your tools, and manages the context for you. You only need to connect Toast and register your tools.

• Point Pi at the Completions API by adding Toast as a custom model in ~/.pi/agent/models.json: an openai-completions provider with baseUrl https://api.mixedbread.com, your MXBAI_API_KEY, a context window of 131072, and max output tokens of 4096. Set the recommended sampling parameters (temperature 0.7, top_p 0.95) and leave thinking off.
• Define your search tools as a Pi extension: pi.registerTool({name, description, parameters, execute}), where parameters is the JSON schema for the arguments. All the guidelines from the Python section apply unchanged: the description is the instruction Toast follows, keep parameters flat and JSON-native, return the JSON envelope with stable chunk_id handles, and return errors as data instead of throwing.
• Pi's built-in compaction summarizes the history. That works, but budget it against the 130k input limit, and prefer prune-style removal of stale tool results where you can.

Example: the tool-call loop over the Completions APILink to section

A minimal harness loop wiring a custom tool (the bm25_search above) to Toast over the Completions API. The tool schema is exactly what a harness would generate from the function's docstring and Annotated hints (see the Python section above) — the raw API always takes the explicit JSON schema. Hosted store tools run server-side; only your own tools come back to you, with finish_reason: "tool_calls":

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.mixedbread.com", api_key=MXBAI_API_KEY)

tools = [
    {
        "type": "function",
        "function": {
            "name": "bm25_search",
            "description": "Keyword-based BM25 search over the corpus. This tool matches "
                           "keywords only — send keyword-heavy queries, not questions. "
                           "Use for rare terms, names, codes, and exact vocabulary; use "
                           "the semantic search tool for meaning-based queries. Returns "
                           "up to top_k chunks with stable chunk_id handles.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "Space-separated keywords, no natural-language "
                                       "questions, no boolean operators. Example: "
                                       "'jordan international goals caps'",
                    },
                    "top_k": {
                        "type": "integer",
                        "description": "Number of chunks to return, max 20.",
                        "default": 5,
                    },
                    "mode": {
                        "type": "string",
                        "enum": ["chunks", "documents"],
                        "default": "chunks",
                    },
                },
                "required": ["query"],
            },
        },
    },
    # bringing your own search backend? disable all five hosted store tools:
    {"type": "store_search", "enabled": False},
    {"type": "store_grep", "enabled": False},
    {"type": "store_list_chunks", "enabled": False},
    {"type": "store_metadata_facets", "enabled": False},
    {"type": "list_stores", "enabled": False},
]

messages = [{"role": "user", "content": "Which suppliers had recalls in 2023?"}]

completion = client.chat.completions.create(
    model="toast-1", messages=messages, tools=tools,
    parallel_tool_calls=True, temperature=0.7, top_p=0.95,
)
choice = completion.choices[0]

while choice.finish_reason == "tool_calls":
    messages.append(choice.message)  # the assistant turn carrying the tool calls
    for call in choice.message.tool_calls:  # may be several — execute all of them
        result = bm25_search(**json.loads(call.function.arguments))
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),  # errors go here as data too
        })
    completion = client.chat.completions.create(
        model="toast-1", messages=messages, tools=tools,
        parallel_tool_calls=True, temperature=0.7, top_p=0.95,
    )
    choice = completion.choices[0]

print(choice.message.content)  # the final answer

Two properties of the loop to keep in mind:

• The API is stateless: every request resends the full message history, and each turn appends the assistant tool-call message plus one tool message per call.
• One turn can carry several parallel calls: answer every tool_call_id, then send the next request.