Context eval

kapi steers model output by injecting context: a terms store that mandates renderings, a voice guide, an instruction. This page measures whether each model actually follows that context — not whether it translates well, which is a different question. The sibling batch eval measures structural integrity and cost; this one measures obedience.

How it is measured

The core metric is a differential. An engineered corpus is translated twice through the production pipeline — once bare (no context) and once steered (terms + voice profile + instruction, injected exactly as production injects them) — and both passes are scored with kapi’s own deterministic check tools (term-check, dnt-check, voice-vocab-check, pattern-check). Two numbers fall out per dimension: absolute adherence (did the steered output satisfy the requirement) and lift (steered minus bare — how much the context moved the model). Lift is the decision-relevant one: a model with high absolute adherence but no lift already “knew” it, and our context earns no credit. A model with high lift is genuinely steerable, which is what context injection is buying.

Every fixture is a trap: a naive translation violates the context. A mandated term whose natural rendering differs from the mandate, a product name that reads like a common noun, casual English tempting an informal register, a source that ends in the exclamation mark the instruction forbids. There are distractors (a lowercase “compass” that is a real compass and must be translated) and declared-winner conflicts (the terms store pins a compound containing a forbidden word — the pin wins, and the scorer knows it). Results are reported per dimension — terminology, voice, instruction — never as one collapsed score: a model can be excellent at terminology and poor at voice, and the collapsed number would hide the thing you would act on.

The deterministic checks are the backbone. The genuinely subjective remainder of voice — register, naturalness, restraint — is scored by a cross-family LLM judge under a fixed yes/no rubric, blind to which model and which variant produced a text, and its scores are published only once judge–human agreement has been measured above a stated bar. Until then, judged numbers stay in the record but off this page.

What the measurement found

The most steerable model measured is haiku ( nb ), which the context moved by +73.1pp overall.Every measured model shows positive lift — the context earns its tokens everywhere.The weakest dimension with context applied is instruction at 97.6% steered adherence — the ceiling context injection currently hits.

Reproducing this

The harness is scripts/contexteval. A run against the built-in demo stub exercises the harness and measures nothing about any model; such runs are marked simulated and are excluded from every chart on this page. Adherence varies by target language, so the published sweep covers more than one.

make context-eval                    # demo stub: proves the harness, measures nothing
make context-eval-publish            # the real sweep → this page's data