Tests and Evals
Results are grouped by what is under test: engine correctness, model output and agent behaviour. Engine tests assert expected results, model evaluations estimate quality, and agent evaluations score individual scenarios. Each group follows the relevant architecture.
7measured
9partial
2unvalidated
16stale or undated
0/6layers without measurements
Engine and formats3 measured3 partial
Subjectkapi's own code: what it does to a document it reads, holds and writes back.EvidenceDeterministic checks compare the content model and output with expected results. Repeated runs use the same inputs.Gates CIYes. A failure here stops a build.
6 evals across F, E · open →
AI and context3 measured4 partial2 unvalidated
SubjectWhat a model writes under kapi's governance, and whether the governance reached it.EvidenceScores estimate performance on the stated corpus. Model-judged scores also depend on the judge's validated agreement with people. Each card reports corpus size to make the sample limitations visible.Gates CINo. Results are tracked over time; regressions require investigation.
9 evals across C, M, A · open →
Agent skills1 measured2 partial
SubjectAn agent driving kapi: the shipped Agent Skill, and the MCP tools an assistant calls.EvidenceRepeated scenarios measure tool selection and whether the agent completes the task with a passing gate. Scores use observed actions and gate results rather than an LLM judge.Gates CINo. Runs use an interactive agent on a maintainer's machine. CI reads the committed dataset; its date indicates when the results were measured.
3 evals across S · open →
What the statuses mean
measuredruns, data committed, numbers can be read as they standpartialruns, and covers less than the layer needsunvalidatedproduces numbers that should not yet be relied onblockedthe harness runs; the surface it measures returns nothingnot measurednothing measures this