Batch eval

kapi groups blocks into a single LLM call to reduce request overhead. This page measures how batch size affects cost, throughput and structural integrity. The evaluation is repeated as models change. The context eval measures whether models follow the terms and writing guidance supplied by kapi.

How it is measured

A fixed corpus is translated at each batch size, and the result is scored for structural integrity: segments must retain their identity, placeholders and tags. A dropped or merged segment, a changed identifier, or a missing placeholder such as {0} can prevent a valid write back to the source format. These checks require no reference translation and can be applied across models and language pairs. They do not assess wording quality. Cost and throughput are recorded from the same runs.

The corpus contains 600 distinct blocks covering ambiguous UI strings, placeholders, inline markup and long prose. Distinct strings prevent a model from passing by copying one segment’s answer into another. Each batch size is tested more than once to account for variation between runs.

What varies with batch size

Measured 2026-07-15 · target de · 600 blocks (181 ui, 118 ui-ambiguous, 120 placeholder, 61 inline-tag, 120 prose) · corpus d46f4b2feac7

The charts below compare cost, structural errors and throughput at each measured batch size. Interpret each curve within its model and provider configuration.

The logarithmic scale makes models with substantially different prices comparable. A falling curve shows fixed request overhead, including the system prompt and JSON schema, spread over more blocks. The size of that overhead varies by provider. A rising curve at large batch sizes can indicate output truncation: kapi splits the batch and retries, adding to the cost.

Structural integrity

Structural errors per 1,000 blocks include dropped, merged or renumbered segments and damaged placeholders or tags. Zero is the expected result. Repeated measurements can reveal regressions even when the provider keeps the same model alias.

Throughput

Source words per second. Comparable within a model across N, not between vendors: the sweeps ran at different concurrencies, and the drop at the right-hand end is the same truncation-and-retry that lifts the cost curve.

What the sweep found

The largest measured batch contains 600 segments. The worst point anywhere in the sweep is 99.1% intact (gemini-3.5-flash at N=256). The comparison below shows whether structural errors become more frequent at larger batch sizes.

Small batches have a higher break rate. Rates are normalized per 1,000 blocks to account for different sample sizes. Small batches (N ≤ 16) broke 2.0 and large ones (N ≥ 128) broke 1.2. In this sweep, larger batches preserved structure at least as well as smaller batches.

Output-token limits can increase retry costs. A blocking call may emit at most 16,000 tokens (NonStreamingMaxOutputTokens: asking a synchronous request for more means holding an HTTP connection open for many minutes). A batch whose reply would exceed that can be truncated into invalid JSON. kapi then halves the batch and translates each half. Retrying avoids accepting the truncated result, but adds to the cost. At the largest measured batch size: eu.anthropic.claude-sonnet-4-6 at N=600 cost 2.5× the tokens and ran 4.2× slower than at N=256; gemini-3.1-flash-lite at N=600 cost 2.1× the tokens and ran 4.3× slower than at N=256; gemini-3.5-flash at N=600 cost 2.2× the tokens and ran 4.6× slower than at N=16.

kapi packs batches against a token budget, with an additional block cap of N≤64. Longer blocks usually reach the token budget first.

What it costs, and what you give up

Every call includes a fixed overhead for the system prompt and JSON schema. Batching spreads this overhead across more blocks. The savings depend on the provider; the Δ column below reports the measured difference.

Batching also reduces the number of requests and can improve throughput. On rate-limited routes, it may be necessary to complete a run. For example, the smallest batch sizes for eu.anthropic.claude-sonnet-4-6 exceeded the account’s Bedrock request quota in this sweep.

Costs are reported in USD per 1,000 source words so models with different tokenizers can be compared on the same content. This harness measures translation. Other batched content tasks may also benefit from reduced request overhead, but their costs require separate measurement.

The smallest batch size measured, against kapi’s shipped ceiling of 64:

Compare the cost and throughput changes with the intact percentage to assess each model's batching tradeoffs.

Interpret these numbers with the following limitations. Bedrock is billed through AWS Marketplace, and the eu. cross-region inference profile is a regional profile: AWS charges exactly 10% more for it than for the otherwise identical global. profile ($3.30/$16.50 per million tokens against $3.00/$15.00). The rates here are the ones actually paid. The claude-code models show no cost, because that sweep reached them through the Claude subscription, which is not billed per token — and whose token counts do not describe an API call either: the CLI wraps every request in its own agent system prompt and reports it as cache creation, so it recorded 240 input tokens across sixty calls. Those counts cannot support a reliable API cost estimate, so the column is blank. Cost for those models requires a sweep against the metered Anthropic API. Speed is not comparable across providers: the sweeps ran at different concurrencies (eu.anthropic.claude-sonnet-4-6: 2, gemini-3.1-flash-lite: 4, gemini-3.5-flash: 4), and the claude-code route pays a CLI session start-up on every call that a direct API call does not. Compare words/sec across batch sizes within the same model. Subscription routes are not billed per token: where a rate is shown, it estimates the cost of the same model on the metered API.

Rates as published 2026-07-14, pinned into each run so a later price change cannot silently restate what a past run cost. Refresh with make update-model-prices.

The numbers

Over time

Intact% at the shipped ceiling (N=64), by run date. A model that regresses while retaining the same alias can be detected by comparing these runs.

Reproducing this

The harness is scripts/batcheval. A run against the built-in demo stub exercises the harness and measures nothing about any model; such runs are marked simulated and are excluded from every chart on this page.

make batch-eval                     # demo stub: proves the harness, measures nothing
make batch-eval-publish             # the real sweep → this page's data