Gå til hovedinnhold

F-03: Identity

Summary​

The framework has four identity primitives, each answering a different question. Entity IDs (core/id) are 8-character base62 strings from crypto/rand (short, URL-safe, dependency-free), used wherever something needs an allocated, opaque handle. Block identity (core/model.BlockIdentity) is a pair of content-addressable hashes that answer "is this the same content?". Reconciled identity (core/reconcile) answers the harder question, "is this the same block as last time?", by grading the two hashes against the previous read rather than by picking a naming scheme, and records the answer as the block's Key. An edition revision (model.EditionRevision) names the content of one edition of a block, so a change or a decision can say which content it was made against. The block-addressed store (core/blockstore) keys on a derived value that namespaces the block by its source file, so blocks from different files never collide.

Context​

IDs appear everywhere: command output, log lines, API responses, internal references. UUID v4 is globally unique and collision-proof, but 36 characters in canonical form are excessive where IDs appear in paths and terminal output, and hyphens make them awkward to read back.

A second problem is specific to content. The framework processes the same content from many sources, so an identifier derived from the content itself, rather than an allocated number, lets identical blocks deduplicate and lets edits be detected by hash comparison. But format readers also assign their own IDs from the source format (XLIFF tu1, tu2, and so on), and those are unique only within one file.

A third problem only appears once a project is iterative. A decision, a translation, and a content-memory entry are all recorded against a block, and the source file is re-read after every edit. Something has to say which block in the new read is which block from the old one, and neither hash can answer that alone.

Decision​

Short base62 IDs​

All entity IDs come from core/id.New():

const base62 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

// rejectThreshold is the largest byte value that divides evenly into the
// alphabet, so each symbol is equally likely: 256 - (256 % 62) = 248.
const rejectThreshold = 256 - (256 % len(base62))

func New() string {
    out := make([]byte, 8)
    var buf [1]byte
    for i := range out {
        for {
            if _, err := rand.Read(buf[:]); err != nil {
                panic("crypto/rand failed: " + err.Error())
            }
            if int(buf[0]) < rejectThreshold {
                out[i] = base62[int(buf[0])%len(base62)]
                break
            }
        }
    }
    return string(out)
}

Properties:

  • 8 characters, base62 alphabet, roughly 47.6 bits of entropy per ID: enough for millions of entities per scope before collision becomes a concern.
  • crypto/rand for cryptographic randomness, with rejection sampling rather than a bare modulo, so no symbol is more likely than another. A modulo over 256 would favour the first eight characters of the alphabet.
  • Zero external dependencies. The implementation does not justify importing a UUID or nanoid library.
  • URL-safe. No special characters, no percent-encoding.

Failing to read crypto/rand panics rather than degrading. A read failure means an OS-level problem, and continuing with weak randomness would silently produce predictable IDs, which is worse than crashing. This matches the standard library's own convention.

Content-addressable block identity​

Blocks carry no allocated entity ID at the framework level. BlockIdentity derives identity from the block itself:

type BlockIdentity struct {
    ContentHash string // SHA-256 of the normalized source text
    ContextHash string // SHA-256 of name, type, and identifying properties
}

func ComputeIdentity(b *Block) *BlockIdentity

ComputeContentHash normalizes by trimming leading and trailing whitespace and nothing else: case and interior spacing are content-significant. Identical strings therefore collapse to one identity and deduplicate across sources. That normalization is part of a wire contract: changing it (adding case folding, say, or Unicode normalization) alters every emitted hash, so a golden-hash test pins the exact output for a known input.

ComputeContextHash folds in the block's name, type, and deterministically-sorted properties, so a structural change is detectable as a context-hash change even when the source text is untouched. Properties whose key begins with @ are advisory and excluded: they are derived locators (a line number, a byte offset), and a locator moves whenever anything above it moves, so hashing one would report an untouched block as changed every time a blank line was added earlier in the file.

The record hash: what a transfer compares​

A third value is derived rather than stored beside the pair. model.ComputeRecordHash folds both halves and the revision of the block's source (model.RunsRevision, which covers the inline codes with their data and attributes), and it is what decides transfer: whether a far side already holds what a block currently is. A revision covers the key it is taken under, so both sides of a transfer take it under the project's source language (venue.RecordHash).

Neither half can decide that alone. The content hash must not move for a reason other than the text moving: it is the identity a decision, a memory entry and a store key are all filed under. But a block is persisted with more than its text: its name, its type, and the properties a reader recorded about it. A reader that starts recording something new leaves the text untouched, so a comparison made on the content hash reports the block unchanged and the new field never arrives. Not on the next transfer: never, because the text is the same forever.

Folding the context half in is what makes an ordinary transfer deliver it, per block, with nothing declared and no version bumped. The cost that would otherwise make this unaffordable (a locator shifting and re-sending a file's whole tail) is already paid for by the advisory prefix above, which keeps derived locators out of the context hash in the first place. The source revision does the same for inline codes: a link whose target moved leaves the text, and so the content hash, as it was, and a far side that never received the new link would go on grading a decision against the old one.

The consequence for readers: a field this hash cannot see is a field that never reaches content already stored elsewhere. Anything a block is persisted with belongs in one half or the other, and anything computed rather than read belongs behind the advisory prefix.

Edition revisions​

A block holds its content as peer editions (F-02), and a change or a decision is about one of them. model.EditionRevision(block, key) names an edition's content: r: and the first 16 hex digits of the SHA-256 of the edition key's canonical text, a zero byte, and the edition's runs as canonical JSON (model.RunsRevision, model.CanonicalRunsJSON). Every key that reaches an edition gives the same revision, and an edition the block does not hold reads as absent (model.AbsentRevision).

What the revision covers is chosen for the questions it answers:

  • Inline codes count. A changed link target or a removed link moves the revision, where the content hash, which hashes trimmed plain text, stays put.
  • Status and origin are left out. A review decision changes them and no word, so it binds to the revision it saw and moves none.
  • Other editions are left out. A derived edition records the revision of the edition it was made from separately, as its derivation (Edition.Derived), so a fix to the source wording does not refuse a concurrent save of a translation. The derivation reads stale once that edition's revision moves (Block.DerivationStanding).

Being a function of content alone, one revision is valid wherever the content is held: a file, a server row or a store. A change set names the revision it read in each operation's if_match and lands only while the edition still has it (E-09). A recorded decision pairs two revisions, the edition it blesses (Revision) and the authoritative edition that edition was made from (Basis), and answers for the block exactly where that pairing holds (C-04). Readers of one document file its source under different keys, so Block.SourceRevisions lists the source's revision under each, and a basis taken under any of them matches.

The revision and the identity pair answer different questions. The content hash and the context hash say which block a read is, and stay the keys sync, the content memory and reconciliation match on; the revision says which content of which edition a write or a decision was made against.

Two producers, two vintages​

Delivering a field to content already stored raises the mirror question: what stops an older producer from taking it away again? A fleet is not one version: a CI runner pinned to an older release, or a laptop that has not upgraded, reads the same files with a reader that records less.

Two mechanisms answer it, and they divide by what can be merged.

Fields are declared, and silence is not deletion. A push declares the property keys its readers emit, computed over every block it read rather than the ones it sent. The far side is authoritative about those keys and leaves the rest of a stored block's properties alone, so a note deleted in the source is deleted here, while a key this producer has never heard of survives. A producer that declares nothing knows nothing: it adds and updates, and removes nothing. No version is involved: a reader that learns to record something new needs no coordination at all.

Structure is versioned, and a downgrade is refused. Segmentation, the run model, overlays, how blocks are named: there is no merging half of one. So the framework states a content-model epoch, the stream records the highest it has received, and a push from a lower one is refused at init, before it uploads anything, rather than applied. kapi push --force carries past it, because a deliberate downgrade is a legitimate thing to want and an accidental one is not. The epoch moves only when produced content gains fidelity an older kapi cannot reproduce from the same file; a new block property is never a reason to move it.

Identity across revisions​

The two hashes answer "is this the same content?" at a point in time. An iterative project needs the second answer, and neither hash can give it alone, nor can a name:

  • Position. A reader without a natural key numbers blocks as it goes (para3, line5, tu2). Delete an earlier block and everything below is renamed, so a decision recorded about one paragraph silently names another.
  • Content. Deriving the name from the text breaks the opposite case: fixing a typo renames the block, so it reads as a deletion plus an addition and loses the history it should have kept. It also collapses the context hash into a restatement of the content hash, destroying the independence the pair exists to provide.

Choosing between them only chooses which way identity breaks. So identity is not a naming scheme. It is the output of matching a read against the previous one, the way rename detection works on files: core/reconcile keeps both signals and grades the pair.

contentcontextmeaningcarries over
matchmatchuntouchedeverything
matchdiffermoved, or the same text reused elsewhereidentity and history
differmatchedited in placeidentity and history; the target is stale
differdiffergenuinely newnothing

The third row is the one no single key can express, and the reason the package exists. The fourth carries nothing over: when both signals change, nothing links the block to its predecessor, and a guess would attach a real decision to the wrong words.

reconcile.Blocks(scope, current, prior) runs four passes (both hashes, then content within the block's own document, then content anywhere in the project, then context alone), and each pass consumes the priors it claims, so a prior (reconcile.Prior) is claimed at most once. Two blocks can never resolve to one key, which is what stops approving one from approving another. Callers may pass the whole project's priors while reconciling one document at a time, which is what lets content moved between files keep its identity, and lets a removed block return to its own history later.

A block's own document is searched before the rest of the project. Short strings repeat across a project, and a change to the context signal fails the exact match for all of them at once. Searched project-wide first, each block would take the first unclaimed prior with its words, often one from another document, and the translation and review attached to that prior would move to a different file. Searched in its own document first, a block keeps its own key, and repeated text inside one document keeps its keys in document order, which is the order a venue serves priors in.

Content is graded before context, and that settles the ambiguous case. Delete a paragraph and the one below slides into the vacated slot: its words match the old lower block, its position matches the deleted one. Content-first reads that as a move, which is what happened; context-first would call it a rewrite and then hand the neighbour's history to whatever arrived next. An edit in place loses nothing to this ordering: its words have changed, so it has no content match to lose, and the context pass still claims it.

Whatever nothing claims is minted from both hashes, so a project with no history reconciles to the same keys on a second run.

Scope, and why documents have identity too​

A block's name is unique only inside its own file: every Markdown document has a para1. Matching on context alone across a project would let a paragraph in one file claim the history of an unrelated paragraph in another, which in a project of a few hundred documents is the common case rather than an edge case. So the context match is scoped to the document: reconcile.Identify(scope, block) records the scope beside the block's own two hashes, unchanged from model.ComputeIdentity, and the pool keys its context lookup on the pair (scope, context hash). The hashes stay the ones the sync wire sends and a venue stores, so a venue's stored hashes serve as a prior set. Content is matched in the block's own document first and then across the project, so a sentence moved between files keeps its translation.

That scope is the document's key rather than its path; otherwise renaming a file would rewrite the context of every block inside it at once, and a file that was only renamed would report every block in it as moved. Documents are therefore matched by the same grading one level up (reconcile.Documents), and their resolved key is what blocks reconcile within.

The pass order there is inverted: same path with the same contents, then same path with changed contents, then a new path with recognisable contents. Path is graded first because a path genuinely identifies a document, whereas para1 does not: two files do not share a path, and a file staying put while its contents change is far more ordinary than a rename. Renames are recognised by content similarity against the prior document at a RenameThreshold of half: enough that a file moved and partly rewritten is still itself, not so little that unrelated files sharing boilerplate merge.

Readers are unaffected by all of this. Their positional names remain exactly what they were and are consumed as the context signal. Reconciliation sits beside the readers rather than inside them so that readers keep reporting what the format actually says.

Where the key lands​

host.ResolveIdentity(byPath, priors) runs the two gradings for a tree of documents: documents first, then each document's blocks against the whole prior set, in a deterministic path order so two runs over the same tree mint the same keys. The resolved key is written onto the block as Block.Key (F-02), and convergence.BlockKey reads it before falling back to the name. The priors are the venue's own tree (venue.Tree.Priors()), fetched before a push declares what it carries, rather than a local store, so a fresh clone resolves exactly as a warm checkout does. A block that matches a prior takes that prior's key: resolving against keys a venue already holds returns them unchanged, only new content is minted, and nothing is ever re-keyed.

Identity across languages​

Everything above answers "is this the same block as before?" within one language. Pairing a source file with its translation asks a different question, and a structural name cannot answer it. Ancestors' text is a legitimate naming input (it is what makes getting-started/install/p readable to the translator who opens the extracted file), but it is written in the document's own language, so the translation names the same paragraph kom-i-gang/installer/p. Matching on the name alone reached only the blocks above the first heading.

Each name therefore has a translation-invariant twin: the same structural path with every segment that came from another block's text replaced by that block's own structural identity. A heading is already addressed by its parent trail plus its ordinal among siblings (precisely so that rewording it moves one signal rather than two), so writing sections that way gives h/h#2/p, which reads the same in every language. Readers compose it beside the name and carry it on the block as model.StructureAnnotation.Address (convergence.BlockAddress). Only formats whose names embed ancestor text compose one; a key path, an element path or a catalog id is already invariant and composes none.

The address is consulted for the one question the name cannot answer, and nothing keyed by the name moves: the context hash, the store key and the XLIFF name attribute all stay on the name. The file-scan pairing (host.OverlayTargets) therefore matches on the name, then on the address, and only then falls back to document position under an equal-block-count guard, which is the last resort for formats that compose no address.

The store key​

Reconciled identity says which block is which. The block-addressed store (core/blockstore) needs a second thing: one string that keys a block and its overlays within a project, stable across re-reads.

That key is neither the bare content hash nor the reader's block ID. Readers assign file-local IDs that restart in every source file, so keying on the raw ID lets blocks and target overlays from different files collide, with the last writer winning. blockstore.StoreKey namespaces the key by the source file's project-relative path:

func StoreKey(sourceRel, blockID, sourceText string) string

The seed is sourceRel plus the block ID, or sourceRel plus the source text when the reader assigned no ID, hashed with the same content-hash function. Every (file, block) pair stays distinct while staying stable across re-reads, so overlays written by a run are found again at merge time.

Both ends of a split run must spell the path identically, which is why blockstore.ProjectRel exists and why it returns an error rather than falling back: the file's path relative to the project root, in exactly the spelling the store's keys use. A run that cannot name its own file the way merge will has nowhere to put its work. The failure mode if the two sides disagree is not an error: merge finds no translation for any block, reads that as pending work, and writes source text into every target file with a zero exit code.

A Store is opened once per process and hands out Session transactions. Blocks are written content-addressed and, once written, are immutable: tools append Overlay layers (targets, annotations, skeletons) keyed by (kind, blockHash) rather than rewriting blocks. This is the substrate flows run against; see C-01: The project model and C-03: The context store and graph.

Operation ids and content addresses​

Every change to a project's context is an operation in the workspace's log (C-11), and an operation has two identities. Its id is time-ordered with a random suffix (workspace.NewOpID), so logs from two machines merge by union and sort into one order; a person types its first characters, the way a commit is named. Its optional content address names what it says: an operation whose address a log already holds is the one already there. A decision's operation (decision.record) carries the ledger entry's own content address (C-04), scoped by project, and the large payloads operations name are blobs addressed by the SHA-256 of their bytes.

Consequences​

  • Paths and command output stay short and readable (aB3xK9mL rather than a 36-character canonical UUID).
  • ID generation needs no external dependency (crypto/rand is the only requirement), and rejection sampling keeps the distribution uniform.
  • Collision probability at roughly 47.6 bits is negligible for the per-scope populations these IDs address, and random IDs avoid the enumeration leaks that sequential IDs produce.
  • Content-addressable BlockIdentity deduplicates identical content across sources and detects edits by hash comparison, with no allocated block ID in the model.
  • Because identity is the output of matching rather than a naming scheme, a reader can keep emitting the positional names its format actually supports, and a paragraph edited in place keeps the decisions recorded against it.
  • Because the store key namespaces by source file, a project can ingest twenty files that each call their first block tu1 without a three-part composite key appearing anywhere else in the stack.

See also​