Batch eval

kapi packs several blocks into a single LLM call, because one call per string is slow and expensive. How many is safe? kapi’s ceiling began as an informed guess; this page is the measurement that replaced it, re-run as models change so it does not quietly go stale. The sibling context eval measures the other axis: whether a model follows the context kapi injects.

How it is measured

A fixed corpus is translated at each batch size, and the result is scored for structural integrity — not wording. The failure that matters when batching is not clumsier phrasing; it is a segment that comes back dropped, merged, renumbered, or stripped of a placeholder or tag. In a translation pipeline that is a correctness failure, because a translation missing its {0} cannot be written back into the source file at all. Scoring structure needs no reference translation, so the same measurement works for any model in any language pair. Cost and throughput are recorded from the same runs.

The corpus is 600 blocks of the things batching is supposed to break — ambiguous UI strings, placeholders, inline markup, and long prose — with every string distinct, so a model cannot pass by copying one segment’s answer into the next. Each size is swept more than once, because a single run of a stochastic model is an anecdote.

What varies with batch size

Measured 2026-07-15 · target de · 600 blocks (181 ui, 118 ui-ambiguous, 120 placeholder, 61 inline-tag, 120 prose) · corpus d46f4b2feac7

Quality does not move with batch size — every model stays structurally intact at every N — so the chart that would show it is a flat line at the ceiling. What moves is what batching costs, so that is the chart.

Log scale, because the models are an order of magnitude apart in price and a linear axis would flatten the cheap ones into the baseline. Two shapes to read. The fall on the left is the per-call overhead — system prompt plus JSON schema, paid once per call — being amortised over more blocks; how steep it is depends on how big that overhead is, which is a fact about the provider, not about batching. On Bedrock that fall is steep — 985 tokens of overhead per call is a lot to repeat. On Gemini it is barely a fall at all: ~106 tokens of overhead leaves almost nothing to amortise, and the curve is flat until it turns up. The rise on the right is the same on every provider, because it is not the provider’s doing: past the output ceiling the reply cannot be emitted in one go, kapi splits the batch and redoes the work, and you pay for the same words twice. Cheapest is in between — and the minimum is broad and shallow, which is the useful part: the exact N barely matters, so long as you are not at either extreme.

Structural integrity, on an axis where a regression would show

Breaks per 1,000 blocks — dropped, merged, renumbered segments, and mangled placeholders or tags. Zero is the expected reading, and the reason to keep drawing it is that a model can regress behind a stable alias without anyone being told. This is the guard, not the headline.

Throughput

Source words per second. Comparable within a model across N, not between vendors: the sweeps ran at different concurrencies, and the drop at the right-hand end is the same truncation-and-retry that lifts the cost curve.

What the sweep found

There is no quality cliff. Batches of 600 segments — most of a document, answered in a single call — came back structurally intact. The worst point anywhere in the sweep is 99.1% intact (gemini-3.5-flash at N=256). Batching is often expected to degrade as N grows, but that did not happen here, and the reason is structural: translating one segment does not depend on having translated the others correctly. The segments are independent, so the failures that do occur are sporadic rather than progressive.

If anything, the small end is worse. These are stochastic models: a sweep this size drops a segment somewhere, so the claim worth making is about the trend, not about perfection. Counted as breaks per 1,000 blocks — a rate, so a batch size is not flattered by having fewer blocks behind it — small batches (N ≤ 16) broke 2.0 and large ones (N ≥ 128) broke 1.2. More calls means more chances to fumble a placeholder, and less surrounding context in which to recognise one — the opposite of the effect the ceiling was set to guard against.

What actually binds is output tokens, and it bills rather than breaks. A blocking call may emit at most 16,000 tokens (NonStreamingMaxOutputTokens: asking a synchronous request for more means holding an HTTP connection open for many minutes). A batch whose reply would exceed that comes back truncated — under a JSON schema, a fragment is invalid JSON — so kapi halves the batch and translates each half. Nothing is corrupted, which is exactly why the integrity line stays at 100%. You simply pay for the work twice. At the top of this sweep that is not a rounding error: eu.anthropic.claude-sonnet-4-6 at N=600 cost 2.5× the tokens and ran 4.2× slower than at N=256; gemini-3.1-flash-lite at N=600 cost 2.1× the tokens and ran 4.3× slower than at N=256; gemini-3.5-flash at N=600 cost 2.2× the tokens and ran 4.6× slower than at N=16.

So the limit that matters is a token budget, not a block count, and that is what kapi packs against. The block cap is a backstop, set at N≤64 — inside the measured-clean range, and few enough that the output budget binds first on anything longer than a short UI string.

What it costs, and what you give up

Every call carries a fixed overhead — the system prompt and the JSON schema that constrains the reply — paid once per call however many blocks ride along. Batching amortises it. How much that is worth depends on how big the overhead is, which varies by provider by an order of magnitude, so the honest answer to “does batching save money?” is it depends on who you call — read the Δ column below rather than trusting a rule of thumb.

What batching buys on every provider is throughput: you wait on a handful of round trips instead of hundreds. And on rate-limited routes it buys the run itself — see eu.anthropic.claude-sonnet-4-6 below, where the smallest batch sizes issue so many calls that the account’s Bedrock quota refuses them outright. There the scarce resource is requests, not tokens.

The unit is USD per 1,000 source words, because content budgets are denominated in words and a token count is a fact about a vendor’s tokenizer rather than about the work. Nothing here is specific to translation: any AI pass over content — a review, a check, terminology, entity extraction — has the same shape and the same economics. Translation is simply the pass this harness drives, because it is the one that batches.

The smallest batch size measured, against kapi’s shipped ceiling of 64:

Read the two Δ columns together against the last one: whatever batching does to your bill, it buys throughput and costs no structural integrity. That is the trade it actually offers.

Three honest caveats about these numbers. Bedrock is billed through AWS Marketplace, and the eu. cross-region inference profile is a regional profile: AWS charges exactly 10% more for it than for the otherwise identical global. profile ($3.30/$16.50 per million tokens against $3.00/$15.00). The rates here are the ones actually paid. The claude-code models show no cost, because that sweep reached them through the Claude subscription, which is not billed per token — and whose token counts do not describe an API call either: the CLI wraps every request in its own agent system prompt and reports it as cache creation, so it recorded 240 input tokens across sixty calls. A dollar figure built on that would be fiction in whichever direction happened to flatter the conclusion, so the column is blank. Cost for those models needs a sweep against the metered Anthropic API. Speed is not comparable across providers: the sweeps ran at different concurrencies (eu.anthropic.claude-sonnet-4-6: 2, gemini-3.1-flash-lite: 4, gemini-3.5-flash: 4), and the claude-code route pays a CLI session start-up on every call that a direct API call does not. Treat words/sec as a within-model comparison across N, not a race between vendors. Subscription routes are not billed per token: for those, the rate shown is what the same model costs on the metered API — the right number to reason with, but not an invoice.

Rates as published 2026-07-14, pinned into each run so a later price change cannot silently restate what a past run cost. Refresh with make update-model-prices.

The numbers

Over time

Intact% at the shipped ceiling (N=64), by run date. A model that silently regresses behind a stable alias shows up here — which is the point of keeping the record rather than measuring once.

Reproducing this

The harness is scripts/batcheval. A run against the built-in demo stub exercises the harness and measures nothing about any model; such runs are marked simulated and are excluded from every chart on this page.

make batch-eval                     # demo stub: proves the harness, measures nothing
make batch-eval-publish             # the real sweep → this page's data