Tests and Evals
Three bands, by what is under test. kapi's own code can be asserted. A model's output can only be estimated. An agent's behaviour is scored scenario by scenario. Inside each band the structure is the architecture.
7measured
9partial
2unvalidated
0 stale or undated
0/6 layers with nothing behind them
Engine and formats3 measured3 partial
Subject kapi's own code: what it does to a document it reads, holds and writes back. Evidence Deterministic. The same document produces the same content model every run, so these evals assert rather than estimate, and a regression is a fact rather than a shift in a distribution. Gates CI Yes. A failure here stops a build.
6 evals across F, E · open →
AI and context3 measured4 partial2 unvalidated
Subject What a model writes under kapi's governance, and whether the governance reached it. Evidence Sampled. A score is an estimate over a corpus, and where a model does the scoring it is an estimate of an estimate. Corpus size is the ceiling on what any of it can conclude, so every card states it. Gates CI No. These are tracked over time, and a drop is a signal to look rather than a build failure.
9 evals across C, M, A · open →
Agent skills1 measured2 partial
Subject An agent driving kapi: the shipped Agent Skill, and the MCP tools an assistant calls. Evidence Scenario-scored. A prompt and a workspace go in, and what the agent did comes out: which tool it reached for, whether it finished at a green gate. No model grades anything. Triggering is stochastic, so a scenario means little until it is repeated. Gates CI No, and these never run in CI. They drive an interactive agent on a maintainer's machine, so the committed dataset is all CI sees and the date on it is the real currency of the numbers.
3 evals across S · open →
What the statuses mean
measuredruns, data committed, numbers can be read as they standpartialruns, and covers less than the layer needsunvalidatedproduces numbers that should not yet be relied onblockedthe harness runs; the surface it measures returns nothingnot measurednothing measures this