AD-017: Bilingual Format Interop

Summary

Kapi ships an end-to-end bilingual round-trip — kapi extract emits a bilingual file for a translator or reviewer, kapi merge applies the returned file back onto the source — with the project's content memory participating on both sides of the loop (pre-fill on extract, absorb on merge). neokapi's native interchange format is the lossless bilingual .kpz (for recipients working in kapi or the standalone neokapi review tool); XLIFF 2.x / PO is the industry-interop tier for third-party CAT tools. Block identity is the merge key; segmentation is an opt-in overlay. The project's .kapi/cache/extractions/<id>/ directory is the source of truth for skeleton bookkeeping so the round-trip is deterministic and portable via git.

Context

Serious translation shops run on bilingual exchange formats. XLIFF 2.x (OASIS; 2.0 in 2014, 2.1 in 2018, 2.2 in 2023) and PO (gettext) are the lingua franca between authored sources, CAT tools (Trados, memoQ, OmegaT, Smartcat, Phrase, Crowdin, Lokalise, Weblate, Poedit, …) and everything else a translation project touches. Kapi already had high-quality format support for bilingual formats — full readers and writers with byte-exact roundtripping (AD-005) — but no workflow glue tying extract, translate, merge, and the content-memory touchpoints on each into one integrated feature.

Every data exchange kapi touches falls into one of six categories:

#	Boundary	Direction	Format(s)
1	Authored source	In	JSON, YAML, HTML, .strings, .properties, .docx, .xml, .md, …
2	Translated output	Out	Same as #1
3	Bilingual exchange	Out/In	Bilingual `.kpz` (neokapi-native, lossless — for a kapi-equipped translator/reviewer); XLIFF 2.x, PO (industry interop); XLIFF 1.2, Qt TS, XLSX-bilingual, SRT, TTML as format support
4	Content memory	In (loop) / Out/In (TMX)	Project content memory (`memory/`), TMX for interop
5	Terminology	In (loop) / Out/In	Project terms store (`terms/`), TBX/CSV/JSON
6	Project portability	Out/In	project folder (`kapi.yaml` recipe + `.kapi/` state)

Boundary 3 is the headline gap this AD closes; boundary 4 is the silent one. The memory exists, TMX ships in and out, but the loop that matters — leverage on extract, absorb on merge — has to be wired or the most valuable asset a translation project accumulates over time goes unused.

Decision

The monolingual sibling: `kapi inspect` / `kapi apply`

extract/merge are the bilingual round-trip — source out, target back. There is a parallel monolingual round-trip on the same content model for source-language work — reviewed edits to a single file in its own format, no target locale: kapi inspect reads a file into anchored content blocks (the read leg) and kapi apply writes a typed change-set of reviewed edits back through the byte-faithful round-trip (the one write verb). The two pairs share the engine — a format reader/writer (AD-005), the block content hash as identity, the skeleton store for faithful reconstruction — but sit on opposite boundaries: extract/merge on boundary 3 (bilingual exchange), inspect/apply on boundaries 1–2 (authored source in, edited source out).

kapi apply is deliberate, reviewed edits; kapi merge is operational memory accretion — keep them distinct. They look superficially similar (both write edits onto source files using the block hash to address blocks), but they answer different questions and must not be unified:

kapi apply lands a human/assistant-reviewed change-set — a content fix, or an asset edit (term, memory pair, brand rule, recipe field) — with a content_hash drift guard and an inline-code fidelity guard, exiting on the gate code when an edit is stale or rejected so a fix loop re-inspects. It is the write half of the inspect/apply loop and never touches a target locale or absorbs into the memory as a side effect. (Adding a pair via an apply entry of kind tm is itself a reviewed edit to the committed .memory.json source, not the automatic merge-time absorb.)
kapi merge applies a translator's returned targets and, by default, absorbs every accepted target segment into the project's content memory (memory-out, below). That accretion is the point of merge and the engine of compounding leverage; it is operational, not a reviewed edit, and is governed by merge.conflict_policy and stale-segment detection rather than a per-block drift guard.

Folding apply into merge would either contaminate the memory with monolingual source edits or strip merge of its absorb behavior; folding merge into apply would lose the conflict policy and the bilingual span mapping. They are separate verbs by design.

CLI surface: extract and merge as top-level commands

kapi extract and kapi merge are top-level kapi commands, not built-in flows dispatched through kapi run. Matches AD-013's existing command tree and keeps discoverability high.

kapi extract -p kapi.yaml                   # all target locales from recipe
kapi extract -p kapi.yaml --target-lang fr  # single target
kapi extract -p kapi.yaml --target-lang fr,de,es
kapi extract -p kapi.yaml --only mobile
kapi extract -p kapi.yaml --pattern 'src/**/*.json'
kapi extract -p kapi.yaml --xliff-version 2.0
kapi extract -p kapi.yaml --format po
kapi extract -p kapi.yaml --no-tm

kapi merge   -p kapi.yaml -i out/myapp-en-to-fr.xliff
kapi merge   -p kapi.yaml -i file1.xliff -i file2.xliff -i file3.po
kapi merge   -p kapi.yaml -i 'vendor-return/*.xliff'
kapi merge   -p kapi.yaml -i vendor-return/
kapi merge   -p kapi.yaml -i ... --no-tm-update

Multi-target in one pass

Both commands handle N target locales in a single invocation. Real projects always ship to multiple languages at once; forcing a loop around a single-target CLI is a non-starter.

Extract: omitting --target-lang defaults to the recipe's target_languages; --target-lang fr,de,es subsets it. One XLIFF / PO per source→target pair (e.g. out/myapp-en-to-fr.xliff), sharing a single extraction batch id / manifest.
Merge: -i is repeatable and accepts file paths, globs, or a directory. A vendor's full N-language return is one invocation. Mixed XLIFF + PO per batch is fine — format is detected per input.
Partial-failure UX: a failure on one pair / input is reported per-item; the rest still apply. Exit code reflects any failure.

Project context auto-discovery

Every project-aware kapi command resolves -p in this order:

Explicit -p <path> flag
KAPI_PROJECT env var
project.ResolveLayout(cwd) — git-style upward walk for the kapi.yaml recipe + adjacent .kapi/ state directory
Fallthrough: one-shot mode (commands that support it) or error "not a kapi project" (commands that require one, e.g. merge)

A directory holds at most one kapi.yaml, so discovery is unambiguous — there is no ambiguous-layout case that requires an explicit -p to resolve.

The helper lives in cli/ once and is reused by run, extract, merge, and any future project-aware command.

Exchange formats

Interchange has two tiers, chosen by who receives the file:

neokapi-native — the bilingual .kpz. A task-scoped profile of the .kpz container (AD-025 §7): one source→target pair, the blocks with faithful inline codes, the segmentation/alignment overlays, the per-source skeleton for round-trip, and the relevant memory-match + term context — one lossless, deterministic, content-addressed file. This is the format kapi distributes to a translator or reviewer working in kapi or the standalone neokapi review tool. It is lossless where XLIFF is lossy, carries memory and term context inline rather than as separate TMX/TBX attachments, and — being Merkle-hashable — gives integrity-verified, diffable review (exactly what changed is visible and tamper-evident). It is ecosystem interchange (both ends need a neokapi reader); making it a cross-vendor standard is an open-spec + second-implementation effort, not a property of the bytes.
Industry interop — XLIFF 2.x / PO. For any recipient on a third-party CAT tool. neokapi stays an excellent citizen of the existing standard: maximally faithful, lossless-as-the-format-allows round-trip. You can never opt out of XLIFF, only offer something better alongside it.

Both tiers flow through the same extract / merge verbs; --format selects the carrier (kpz native, xliff / po interop). kapi extract emits XLIFF 2.2 by default for safe interop with any recipient; the bilingual .kpz is selected with --format kpz for recipients working in kapi or the neokapi review tool.

XLIFF 2.x is the default industry-interop carrier. The reader accepts all three 2.x namespaces as a compatible family (…:2.0, …:2.1, …:2.2) and preserves unknown 2.x attributes round-trip via the layer property map. The writer emits 2.2 by default, with --xliff-version 2.0|2.1|2.2 as an opt-out for consumers stuck on older tooling.

PO (gettext) is the day-one alternate. Selected with --format po. One entry per segment span (the whole block when unsegmented). Kapi-specific bookkeeping rides in developer comments (#. kapi-block: <hash>/<sN>).

XLIFF 1.2, Qt TS, XLSX-bilingual, SRT, and TTML remain available as format support (byte-exact roundtrip via their readers/writers) but do not get extract/merge integration in v1.

Block is the merge key; segmentation is an overlay

The merge key is the block content hash (core/model/identity.go ComputeContentHash, SHA-256 over the normalized source runs). Segmentation is a stand-off overlay over those runs (AD-002), not a rewrite of them, so it never changes the hash — a block's identity is stable across segmentation on/off toggles between extractions.

With no segmentation overlay, a Block emits one XLIFF <segment> / one PO entry over its whole content. When the recipe sets segmentation.source: true, the segment annotator runs as a pipeline stage and attaches a segmentation overlay with stable span ids (s1, s2, …); the writer materializes one <segment> / <ignorable> (or PO entry) per span and gap. Merge maps each returned target back to its source span via the alignment overlay and splices the target runs into place — the block hash is the join key, so a project can flip segmentation on or off between extractions without breaking the memory, QA overlays, or manifest bookkeeping.

Per-segment memory lookup is the matching widening: memory.Lookup keys on the whole block when there is no segmentation overlay. When one is present, extract iterates its spans and looks each up independently for sentence-level leverage via the LookupSegment method on the ContentMemory interface.

Skeleton portability (project-state only, v1)

V1 stores skeletons in project state (.kapi/cache/extractions/<id>/). Merge therefore requires the same .kapi project that produced the extraction. The emitted XLIFF / PO is clean, CAT-friendly, and small.

The door is left open for a future --embed-skeleton flag on kapi extract that would produce a self-contained XLIFF 2.0 with the skeleton embedded; data structures don't assume project-state forever.

Extraction manifest

Each kapi extract run writes a manifest at .kapi/cache/extractions/<batch-id>/manifest.yaml:

schemaVersion: 1
kind: kapi-extraction
batchId: 6f2e8a1c-...
generator: { id: kapi, version: v1.x }
createdAt: 2026-04-24T10:00:00Z
sourceLocale: en
options: { format: xliff2, xliffVersion: "2.2", noMemory: false }
pairs:
  - targetLocale: fr
    output: out/myapp-en-to-fr.xliff
    files:
      - source: src/locales/en/app.json
        sourceHash: sha256:...
        blocks: 412
        leverage: { exact: 108, fuzzy: 67, new: 237 }
        skeleton: skel-<source-hash>.bin
  - targetLocale: de
    output: out/myapp-en-to-de.xliff
    files: [...]

The batch id is stamped in each emitted XLIFF / PO so merge can resolve any returning file back to the right manifest without guessing from file name.

XLIFF 2: <file>-level <notes> entry with category kapi, content batch:<uuid>.
PO: file-header extracted comment #. kapi-batch: <uuid>.

Sub-threshold memory matches are written to .kapi/cache/extractions/<batch-id>/suggestions.jsonl for later analysis without touching the emitted target.

Memory-in on extract (v1, on by default)

kapi extract queries the project's content memory for every segment it emits.

Exact match → pre-fill <target> with state="translated".
Fuzzy match ≥ tm.fuzzy_threshold (default 75) → pre-fill with state="fuzzy" and a matchQuality sub-state attribute.
Ambiguous match (several full-score exacts with differing targets — Match.Ambiguous) → never pre-filled. An unattended merge would turn an arbitrary pick into published content; left empty, the segment surfaces as untranslated for a human to decide.
Sub-threshold matches → suggestions.jsonl, not inlined.

Disable with --no-tm. Additional read-only memories can be declared in the recipe via tm.read: [path, …] and are consulted alongside the project's own memory during pre-fill.

Memory-out on merge (v1, on by default)

kapi merge writes every accepted target segment into the project's content memory. Entries carry provenance:

Origin.Source = "merge"
Origin.Reference = <batch-id>
Origin.Key = <source-file-path>
Block-hash and originating XLIFF filename as properties

Disable with --no-tm-update. Combined with memory-in on the next extract, this is what makes the memory a memory — without the write-back, leverage decays to zero.

Write-back only to the project's own memory. Imported read-only memories (tm.read) are never written to; this keeps imported TMX reproducible from its source.

Conflict policy

merge.conflict_policy (project recipe field) governs:

Applying the translator's target to the source file when an existing target is present on disk;
Writing back to the memory when an entry already carries a translation.

Values:

translator-wins (default) — the translator's target always replaces the existing one.
existing-wins — the existing on-disk / in-memory target is preserved; the translator's target is skipped with a warning.
newest-wins — compare timestamps (file mtime / entry UpdatedAt) and pick the newer.

No interactive prompting — keeps merge scriptable in CI.

Stale segment handling

Merge detects stale segments by comparing the incoming XLIFF's recorded source hash (captured at extract time) against the current source. Stale segments are reported, not silently applied, and not absorbed into the memory even when the conflict policy would otherwise accept the target.

Partial returns are fine: merge finds the extraction manifest by batch id and applies translated segments, leaving untranslated ones alone.

Recipe schema

Three new sections on Defaults:

defaults:
  source_language: en
  target_languages: [fr, de, es]
  merge:
    conflict_policy: translator-wins # | existing-wins | newest-wins
  memory: # content memory (recipe key retained as `tm`)
    fuzzy_threshold: 75 # int 0..100
    read: # optional read-only memories
      - /path/to/corporate.tmx
  segmentation:
    source: false # opt-in
    srx: rules.srx # optional SRX override

Unknown fields are rejected with a clear error. Enum values are validated against the allowed set. Defaults apply when the section is absent.

Relationships to other ADs

AD-005 (Format System) — XLIFF 2.x writer/reader and PO reader/writer are the carriers; skeleton store is how byte-exact roundtrip works.
AD-008 (Project Model) — extract adds the extraction manifest under .kapi/cache/extractions/<id>/. Conflict policy is a new Defaults.Merge section. Auto-discovery uses the existing project.ResolveLayout entry point.
AD-009 (Content memory) — Lookup becomes load-bearing for extract pre-fill. A LookupSegment method is added for per-span matching when a segmentation overlay is present. Merge extends the Origin provenance story with a "merge" source and the batch id in Reference.
AD-010 (Terminology) — term hints on extract, drawn from the terms store, are a natural follow-up (analogous to memory pre-fill). Not in v1.
AD-013 (Kapi CLI) — adds extract and merge top-level commands and describes the auto-discovery resolution order.
AD-024 (Agent Skills) — defines the monolingual kapi inspect / kapi apply loop that is the source-side sibling of extract/merge, the typed change-set, and the content-edit drift / inline-code guards.
AD-015 (Testing & Documentation) — extract/merge each ship with an interactive walkthrough embed and a unified workflow guide on the docs site.

Rationale

Why extract/merge as top-level commands rather than flows? Discoverability. An engineer looking at kapi --help sees them immediately alongside run. Building them as flows would hide them a layer deeper and make their specialized flag shapes (--only, --pattern, -i repeatable, conflict policy) awkward to express.

Why block hash as merge key, not segment position? Segmentation is a stand-off overlay, not a structural change. The same Block may emit 1 or N <segment>s depending on the recipe; using block hash as the stable key means the project can change segmentation settings between extractions without breaking merge. Segment span ids (s1, s2, …) ride inside the segmentation overlay.

Why project-state skeletons (not embedded)? Keeps the emitted XLIFF small and CAT-friendly. Makes the memory absorb cheap (no skeleton to unpack on merge). The project folder is already the unit of portability — git push ships it. Embedded skeleton remains an opt-in escape hatch for workflows that can't ship the project.

Why memory-in + memory-out on by default? Without the loop, the memory is a sidecar — interesting, not load-bearing. The moment the loop is on, each merge makes the next extract cheaper. Making it opt-in would leave most users' memories empty or stale.

Why write-back only to the project's own memory? Imported TMX should be reproducible from its source file; writing into it turns it into a living artifact that drifts from the TMX on disk. The project's own memory is the editable thing.

Summary​

Context​

Decision​

The monolingual sibling: kapi inspect / kapi apply​

CLI surface: extract and merge as top-level commands​

Multi-target in one pass​

Project context auto-discovery​

Exchange formats​

Block is the merge key; segmentation is an overlay​

Skeleton portability (project-state only, v1)​

Extraction manifest​

Memory-in on extract (v1, on by default)​

Memory-out on merge (v1, on by default)​

Conflict policy​

Stale segment handling​

Recipe schema​

Relationships to other ADs​

Rationale​