Tests and Evals
Results are grouped by what is under test: engine correctness, model output and agent behaviour. Engine tests assert expected results, model evaluations estimate quality, and agent evaluations score individual scenarios. Each group follows the relevant architecture.
7measured
9partial
2unvalidated
16 stale or undated
0/6 layers without measurements
Engine and formats3 measured3 partial
Subject kapi's own code: what it does to a document it reads, holds and writes back. Evidence Deterministic checks compare the content model and output with expected results. Repeated runs use the same inputs. Gates CI Yes. A failure here stops a build.
6 evals across F, E · open →
AI and context3 measured4 partial2 unvalidated
Subject What a model writes under kapi's governance, and whether the governance reached it. Evidence Scores estimate performance on the stated corpus. Model-judged scores also depend on the judge's validated agreement with people. Each card reports corpus size to make the sample limitations visible. Gates CI No. Results are tracked over time; regressions require investigation.
9 evals across C, M, A · open →
Agent skills1 measured2 partial
Subject An agent driving kapi: the shipped Agent Skill, and the MCP tools an assistant calls. Evidence Repeated scenarios measure tool selection and whether the agent completes the task with a passing gate. Scores use observed actions and gate results rather than an LLM judge. Gates CI No. Runs use an interactive agent on a maintainer's machine. CI reads the committed dataset; its date indicates when the results were measured.
3 evals across S · open →
What the statuses mean
measuredruns, data committed, numbers can be read as they standpartialruns, and covers less than the layer needsunvalidatedproduces numbers that should not yet be relied onblockedthe harness runs; the surface it measures returns nothingnot measurednothing measures this