Build Your Own Harness

Toast is a deep search and lookup agent, trained end-to-end to gather evidence and to answer from it: as a ranked evidence list, as ranked evidence plus an answer, or as a plain-text answer. You can use Toast with our Toast 1 harness directly via our Search API with agentic search feature, or let the API run the search loop for you with the hosted store tools.

Like most agents, Toast is only as good as its tools. It performs best with a late-interaction style semantic search, but has no issues working with your other search backends. The search agent adapts well to new tools and custom harnesses: file search tools, tools to access domain knowledge, or custom answer tools. This guide collects recommendations and guidance for defining custom harnesses with our Completions API so you can make best use of Toast 1's strengths for your use case.

Toast 1 is trained for deep evidence gathering and for answering from what it found. It prefers to fan out searches via parallel search calls, resulting in high exploration for minimal latency. It's the perfect multilingual exploration agent for knowledge work: a substitute for GPT-5.6 Luna/Terra or Claude Haiku/Sonnet as a look-up or grounded question-answering subagent. If these characteristics match your use case, follow this guide to create your own fast searcher.

Platform constraints

• Toast has thinking disabled. It was trained without thinking and is only served without thinking to have state-of-the-art retrieval performance at minimum latency.
• Sampling parameters are configurable; we recommend temperature 0.7, top_p 0.95.
• Toast is served at seq_len = 131072: every input message needs to stay below 130k tokens.
• Generation defaults to 4096 tokens per completion; raise max_completion_tokens if you need more.

Context management comes in two modes:

• Pass-through (the default): manage and compact your context against the 130k input limit yourself (for example, use it for a Pi harness internal compaction). The API adds nothing and edits nothing; an oversized request is an honest 422 context_length_exceeded, never a silent truncation.
• Declared context_management: in our Toast 1 harness, Toast uses pruning as its context management: a prune_context tool the agent calls to remove chunks it considers irrelevant, invoked as it approaches the context limit. The API offers the same mechanism as an opt-in: declare {"context_management": {"edits": [{"type": "prune_context"}]}} and the server offers the model the harness's prune_context tool over every tool result, yours included (details). Prefer declaring it over hand-rolling your own pruning; our inference and latency are optimized for prune-style removal of stale tool results over summarizing compaction.

Do not use previous_completion_id after your application has edited the history; it only works when the history you resend extends the stored one. After client-side pruning, just resend the full edited messages list (see the loop example at the end of this guide). Server-side edits are different: prunes applied under declared context_management never invalidate previous_completion_id, because the server holds the authoritative edited context.

Harness design & Completions API guidance

The Completions API is an OpenAI-compatible chat completions endpoint: provide your prompt, messages, tool definitions, and sampling params exactly the way you would with the standard OpenAI client. You send a messages list plus your tool schemas in tools, and Toast answers with either text or tool calls. The API is stateless unless you use previous_completion_id: every request resends the full message history, and the server executes none of your tools. When Toast calls one of your tools, the completion ends with finish_reason: "tool_calls"; your harness executes the calls, appends one tool message per call, and sends the next request. That loop is your whole harness.

The API adds nothing on its own: no injected prompt, no injected tools, one generation per request. The one server-side feature this guide recommends is declared context_management (see Platform constraints above), strictly opt-in.

The following guidelines are design choices inspired by our Toast 1 harness. Treat them as recommendations, not requirements.

• Toast is trained to work within a bounded number of turns and a bounded number of parallel tool calls. Communicate those boundaries in the harness. Our Toast 1 harness defaults to 4 turns with up to 8 parallel calls per turn, but this is a soft recommendation; Toast can easily go up to 8, 12, or more turns.
• For similar retrieval performance, enable parallel tool calls in your harness and make your tools safe to execute concurrently. Parallel tool latency is bounded by the slowest tool you include.
• Do not overflow the 130k token context limit when providing the full results of many parallel calls back to the agent. With up to 8 parallel calls per turn, uncapped tool returns fill the context quickly. If your tools return docs with many tokens, truncate them.
• Return short stable handles with your search tools' results, like a short chunk_id. Every result your search tools return needs a stable id, and any tool that takes other tools' outputs as inputs must accept exactly the ids you emitted. Tools that operate on retrieved results are easy to define this way.
• Deduplicate if you want: filtering already seen search results out of the tool call results leads to less context rot and higher recall.
• Give Toast a terminal: a tool to end the searching and report its findings, or a plain-text answer turn. Our Toast 1 harness uses a submit_ranking tool that takes the chunk_ids your search tools emitted plus a relevance score per chunk and a short ranking_strategy (see Terminal modes below).
• Define your tools to return a dict, not prose text. Our Toast 1 harness tool results are JSON envelopes: {"query": ..., "results": [...], "candidate_count": N} with per-result entries like {"chunk_id": ..., "text": ..., "score": ..., "metadata": {...}}. Prefer returning a similar shape from your functions.
• Communicate tool schema errors back to Toast instead of raising them. If you provide the tool error in the tool response, Toast can retry the call in the next turn; we trained Toast to recover well from structured errors. Example: catch the exception and return {"error": "date must be ISO format, got '3/5/24'"}.

Terminal modes

Toast was trained on three terminal modes (three ways to end a run). Pick the one that matches what your application consumes and keep your wording close to ours:

• submit_ranking, ranking only (our harness default): Toast ends with a submit_ranking call carrying chunks (chunk_id plus relevance_score) and a ranking_strategy; your application generates the answer from the ranked chunks.
• submit_ranking with answer: add a required answer string to the reporting tool. Our parameter description: "Your final answer to the original user query, based only on retrieved evidence. Required on every submit_ranking call: give your single best answer even when uncertain; if the evidence is insufficient to answer, say so." Toast returns evidence and answer in one structured call.
• Plain-text answer: offer no reporting tool at all and leave tool_choice on auto. Instruct Toast that every response must contain tool calls until it answers, and that a plain-text reply with no tool calls ends the run: "Do not report chunk lists or rankings; deliver the answer itself." The completion's content with finish_reason: "stop" is the answer.

In every mode, tell Toast to base the answer only on retrieved evidence and to say so when the evidence is insufficient. When the round budget runs out, force the terminal turn with a short user message ("You have reached the search limit. Do NOT search further. ...").

Phrasing of instructions

Toast performs best writing human-style questions for a semantic search tool and writing regex for grep tools. Be precise in your custom tool definitions and give clear instructions on how to use the tool (see the example at the end of this guide).

When adding custom search tools, providing instructions on how to formulate queries helped a lot to boost retrieval performance. An example for a BM25 tool: "this BM25 tool is matching keywords, write only keyword-heavy queries".

Build your own harness with Python

For a custom searcher harness in Python, write the tool descriptions directly in the docstrings and type hints of your tool functions. Docstrings and type hints are the interface Toast sees: generate the JSON tool schema from them with any docstring-based schema helper: the docstring becomes the tool description, and Annotated strings become the parameter descriptions.

• Toast follows tool descriptions as instructions. Say what the tool matches on, when to prefer it, and how to phrase input ("this tool matches keywords only. Send keyword-heavy queries, not questions").
• Keep tool signatures flat and JSON-native. Toast was trained on flat argument objects: str, float, bool, list[str], and one level of typed dicts.
• Prefer snake_case names. Mark what's essential as required and give defaults for the rest of your parameters.
• Use Literal/Enum for closed choices. Trained tools express modes as enums (filter_mode: Literal["all", "any"], direction: Literal["asc", "desc"]).

Example 1: a simple semantic search tool with just a query and a top_k parameter.

from typing import Annotated


def semantic_search(
    query: Annotated[str, "Natural-language query for a single search aspect; "
                          "avoid Boolean syntax, regex, and keyword dumps."],
    top_k: Annotated[int, "Number of chunks to return, max 20."] = 5,
) -> dict:
    """Execute a meaning-based semantic search query over the corpus and return the
    most relevant chunks. Use natural language; phrase queries as human-style
    questions. Do not use for keyword, regex, or literal-string matching.
    Returns up to top_k chunks with stable chunk_id handles."""
    hits = my_search_backend(query, top_k=top_k)  # your implementation
    return {
        "query": query,
        "candidate_count": len(hits),
        "results": [
            {"chunk_id": h.id,                   # short stable handle, e.g. "c12"
             "score": round(h.score, 4),
             "text": h.text,                     # clipped, not the whole document
             "metadata": h.metadata}
            for h in hits
        ],
    }

Example 2: a custom BM25 keyword-search tool.

from typing import Annotated, Literal


def bm25_search(
    query: Annotated[str, "Space-separated keywords, no natural-language questions, "
                          "no boolean operators. Example: 'jordan international goals caps'"],
    top_k: Annotated[int, "Number of chunks to return, max 20."] = 5,
    mode: Literal["chunks", "documents"] = "chunks",
) -> dict:
    """Keyword-based BM25 search over the corpus. This tool matches keywords only.
    Send keyword-heavy queries, not questions. Use for rare terms, names, codes,
    and exact vocabulary; Returns up to top_k chunks with stable chunk_id handles."""
    hits = bm25_index.search(query, k=top_k, mode=mode)  # your implementation
    return {
        "query": query,
        "candidate_count": len(hits),
        "results": [
            {"chunk_id": h.id,                   # short stable handle, e.g. "c12"
             "score": round(h.score, 4),
             "text": h.text,                     # clipped, not the whole document
             "metadata": h.metadata}
            for h in hits
        ],
    }

Note how the description tells Toast what the tool is for, how to phrase input, when not to use it (and which tool to use instead), and how many results to expect. The return value is a JSON envelope with short stable handles and a score.

Build your own harness with Pi

If you use Pi as your agent framework, the harness loop already exists: Pi runs the tool-call loop, executes your tools, and manages the context for you. You only need to connect Toast and register your tools.

• Point Pi at the Completions API by adding Toast as a custom model in ~/.pi/agent/models.json: an openai-completions provider with baseUrl https://api.mixedbread.com/v1, your MXBAI_API_KEY, a context window of 131072, and max output tokens of 4096. Set the recommended sampling parameters (temperature 0.7, top_p 0.95) and leave thinking off.
• Define your search tools as a Pi extension: pi.registerTool({name, description, parameters, execute}), where parameters is the JSON schema for the arguments. All the guidelines from the Python section apply unchanged: the description is the instruction Toast follows, keep parameters flat and JSON-native, return the JSON envelope with stable chunk_id handles, and return errors as data instead of throwing.
• Pi's built-in compaction summarizes the history. That works, but budget it against the 130k input limit, and prefer prune-style removal of stale tool results where you can.

Example: the tool-call loop over the Completions API

A minimal harness loop wiring a custom tool (the bm25_search above) to Toast over the Completions API. The tool schema is exactly what a harness would generate from the function's docstring and Annotated hints (see the Python section above); the raw API always takes the explicit JSON schema. Your own tools come back to you, with finish_reason: "tool_calls":

import json
from openai import OpenAI

client = OpenAI(base_url="https://api.mixedbread.com/v1", api_key=MXBAI_API_KEY)

tools = [
    {
        "type": "function",
        "function": {
            "name": "bm25_search",
            "description": "Keyword-based BM25 search over the corpus. This tool matches "
                           "keywords only. Send keyword-heavy queries, not questions. "
                           "Use for rare terms, names, codes, and exact vocabulary. "
                           "Returns up to top_k chunks with stable chunk_id handles.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "Space-separated keywords, no natural-language "
                                       "questions, no boolean operators. Example: "
                                       "'jordan international goals caps'",
                    },
                    "top_k": {
                        "type": "integer",
                        "description": "Number of chunks to return, max 20.",
                        "default": 5,
                    },
                    "mode": {
                        "type": "string",
                        "enum": ["chunks", "documents"],
                        "default": "chunks",
                    },
                },
                "required": ["query"],
            },
        },
    },
]

messages = [{"role": "user", "content": "Which suppliers had recalls in 2023?"}]

completion = client.chat.completions.create(
    model="toast-1", messages=messages, tools=tools,
    parallel_tool_calls=True, temperature=0.7, top_p=0.95,
)
choice = completion.choices[0]

while choice.finish_reason == "tool_calls":
    messages.append(choice.message)  # the assistant turn carrying the tool calls
    for call in choice.message.tool_calls:  # may be several; execute all of them
        result = bm25_search(**json.loads(call.function.arguments))
        messages.append({
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(result),  # errors go here as data too
        })
    completion = client.chat.completions.create(
        model="toast-1", messages=messages, tools=tools,
        parallel_tool_calls=True, temperature=0.7, top_p=0.95,
    )
    choice = completion.choices[0]

print(choice.message.content)  # the final answer

Three properties of the loop to keep in mind:

• The API is stateless: every request resends the full message history, and each turn appends the assistant tool-call message plus one tool message per call. The next section removes the resend.
• One turn can carry several parallel calls: answer every tool_call_id, then send the next request.
• The loop is deliberately minimal: no round bound, and it ends on the plain-text terminal. In your harness, cap the rounds and, once the cap is hit, force the terminal turn as described under Terminal modes.

Continue a stored completion

The loop above resends the full message history on every request. With store left at its default of true, the API stores each completion, and the next request can continue it instead: name the completion in previous_completion_id and send only the new messages. The server restores the stored model context, hosted tool calls and server-side context edits included, and appends your messages to it.

The loop changes in two places: messages shrinks to the new tool results, and each request names the completion it continues.

completion = client.chat.completions.create(
    model="toast-1", messages=messages, tools=tools,
    parallel_tool_calls=True, temperature=0.7, top_p=0.95,
    extra_body={"context_management": {"edits": [{"type": "prune_context"}]}},
)
choice = completion.choices[0]

while choice.finish_reason == "tool_calls":
    results = [
        {
            "role": "tool",
            "tool_call_id": call.id,
            "content": json.dumps(bm25_search(**json.loads(call.function.arguments))),
        }
        for call in choice.message.tool_calls
    ]
    completion = client.chat.completions.create(
        model="toast-1",
        messages=results,  # only the new messages; the server holds the rest
        tools=tools,
        parallel_tool_calls=True, temperature=0.7, top_p=0.95,
        extra_body={
            "context_management": {"edits": [{"type": "prune_context"}]},
            "previous_completion_id": completion.id,
        },
    )
    choice = completion.choices[0]

print(choice.message.content)  # the final answer

• Do not resend the assistant turn: the stored completion already contains it. Your tool messages answer its tool_call_ids.
• The server stores the conversation, not the request configuration: tools, sampling parameters and context_management go with every request, as in the stateless loop.
• This is where declared context_management pays off across turns: prunes Toast applied in an earlier request stay applied, and each usage.prompt_tokens reflects them. A full-history resend rebuilds the context from your messages, and pruned content comes back.
• previous_completion_id requires the stored history to be extended, never edited (see Platform constraints above). After your application edits the history, resend the full messages list without it.
• Completions in a chain group into one conversation for listing and deletion.