Context eval

kapi supplies models with project context, including required terms, a voice guide and instructions. This page measures how that context affects adherence to each requirement. It does not assess overall translation quality. The batch eval measures structural integrity and cost.

How it is measured

A test corpus is translated twice through the production pipeline: once bare (without context) and once steered (with terms, a voice profile and instructions). Both passes are scored with kapi’s deterministic checks: term-check, dnt-check, voice-vocab-check and pattern-check. Each dimension reports absolute adherence (the proportion of requirements satisfied with context) and lift (adherence with context minus adherence without it). High adherence with no lift means the model already met those requirements without the supplied context. Positive lift measures the improvement attributable to that context in this corpus.

Fixtures test requirements a translation might otherwise violate: an unusual required term, a product name that resembles a common noun, a formal register for casual source text, or a ban on exclamation marks. Distractors test whether the model applies rules too broadly. Conflicting rules have an explicit priority; for example, a required compound can take precedence over a ban on one of its words. Results are reported separately for terminology, voice and instructions so strengths in one dimension do not obscure weaknesses in another.

Subjective voice qualities, including register, naturalness and restraint, are scored by an LLM judge from a different model family. The judge uses a fixed yes/no rubric and receives no information about the model or variant that produced the text. Scores are published only after measured agreement with human reviewers exceeds the stated threshold. Until then, they remain in the run record but are excluded from this page.

What the measurement found

The model with the greatest measured lift is haiku ( nb ), with an overall improvement of +73.1pp.Every measured model shows positive lift on this corpus.The weakest dimension with context applied is instruction at 97.6% steered adherence.

Reproducing this

The harness is scripts/contexteval. A run against the built-in demo stub exercises the harness and measures nothing about any model; such runs are marked simulated and are excluded from every chart on this page. Adherence varies by target language, so the published sweep covers more than one.

make context-eval                    # demo stub: proves the harness, measures nothing
make context-eval-publish            # the real sweep → this page's data