Gå til hovedinnhold

The file home: core/change/filehome

The file home is the home of the change service (E-09) for documents kept as files in a working tree. This note covers how it locates a document, how a stage reads and writes it, how it adds and removes blocks, how it treats a bilingual file, how a commit settles against other writers, how editions in files of their own, and editions the workspace home keeps until their file exists, are joined and written, how it commits a document a run produced whole, and the tests that hold it to the contract.

Locating a document​

filehome.New(layout, options) builds the home over a Layout, which resolves a reference into a filehome.Doc: the canonical reference, the file on disk (or the archive and its member for container!entry), a Binding that opens the format's reader and writer each configured for the document, the source locale and encoding, whether the format holds editions in the file, and a function that names the file of each edition kept in a file of its own.

filehome.DirLayout serves a directory with no recipe: references resolve under its root with filehome.ResolvePath, which refuses a path that leaves the root and confines an archive member to an archive under it, and the format is the one detection finds. The kapi host supplies a project layout (host/changes.go) that reads each source file with the format and configuration its content item declares, maps each target file back to the source and locale it holds, and names a source's target files through the recipe's target template. A reference to a source file resolves on its own, through the rule that names the item claiming a path (ProjectContext.ResolvePaths), so locating it costs the same in any size of project; the recipe's content patterns are expanded once per service, and only for a reference that names no source file, such as the file of a translation. The layout's source locale is the one its caller names (ChangeServiceOptions.SourceLocale), else the recipe's: the MCP server resolves it per call, because one server serves several projects. A source code file whose comments kapi reads, and that no format reads, is refused as unsupported with the capability comment.

A stage​

Session.Stage makes one pass over the document:

  1. It hashes the file (sha256: over the bytes on disk; for an archive member, over the whole archive), and then each file of an edition the change addresses that lives in a file of its own. Every file is hashed before it is read, so a writer that commits between the read and the commit leaves the file at another digest than the one recorded.
  2. When the change adds or removes blocks, it has the format's writer write those edits first, and the rest of the pass reads the result (see Adding and removing blocks).
  3. It joins those editions to the document (see below).
  4. It reads the document through the binding's reader with the writer's skeleton store wired, the read every read path of the service makes, so the blocks an edit is addressed from are the blocks the write sees. A read that writes nothing wires a store that keeps nothing (format.NewWiredReadSkeleton), so it creates no file. A block the reader left with no language gets the document's, so its own edition answers to that language as well as to the empty edition key.
  5. It passes every block to the service's editor, which applies the operations addressed to it and reports the editions it changed.
  6. When the change writes the document's own file, it writes the result through the same format's writer into a temporary file beside the document (atomicfile.StageWithParents), with the mode the document has and the symlink resolved, hashing the bytes as they are written. A writer that re-reads the original gets the document's path; one that needs its bytes gets them.

When the reader and the writer both stream (format.StreamingReader and format.StreamingWriter) and the writer needs no copy of the original, the read and the write run concurrently through a streaming skeleton store, so neither the input nor the block stream is held whole. Any other pair reads every part and then writes them. The editor keeps only the blocks it changed. A pass that changes nothing in the document's own edition discards the temporary file and reports the file as read.

The service compares each changed edition with its staged file: an edition whose file the stage leaves byte for byte as it was is one the format has no place for there, and its operation is refused as unsupported rather than reported applied.

An archive member is read from its bytes, written to a buffer, and spliced into a copy of the archive (container.Transform) staged beside it, every other member copied as it was. Two members of one archive in one change set are refused as invalid: each would stage a copy of the archive from what it read.

A bilingual file​

A format that holds its translations in the document, such as PO, XLIFF or TMX, keeps one translation per language, so an edition with a tone or a channel has no place in it and is refused as unsupported. Some readers model a translation only when told its language: a PO catalog's msgstr reads as the French edition only when the reader is given fr. The layout gives it as Doc.TargetLocale (the host takes --target-lang), and every pass over the file hands it to the reader and to the writer. Without it the catalog's translation is not an edition a read shows, and an edit of it changes no byte, so it is refused. In a project, a bilingual source whose collection names a target holds no translation of its own: the host's layout gives it one file per edition and no TargetLocale, whatever language its caller was told, so the French of po/en.po, or of the template po/messages.pot, with target po/{lang}.po is read from and written to po/fr.po.

A stage that removes a translation the file holds reads the staged bytes back through the format's reader (verifyRemoved) and refuses the removal as unsupported with capability remove_edition when the translation is still there. A translation kept in a bilingual file of its own (fr/messages.po, or x/fr.xlf under a target template) is removed from its partner block there and read back the same way, in the edition's language (verifyEditionRemoved). A PO catalog writes the entry with an empty msgstr, which reads as no translation; the XLIFF 1.2, XLIFF 2 and TMX writers, which follow the Okapi filters, write a translation of every unit and take the source where the block holds none, so a removal there is refused rather than write the source in the translation's place. Their description lists remove_edition for each block all the same, because change.BaseOps gives it to every interchange format and no writer declares that it cannot remove one. An archive member is not read back, and neither is a flow's output: kapi exec remove-target writing an XLIFF translation through the writer gets the source in the removed translation's place.

A commit​

Staged.LockKeys names the lock file of every file the stage reads or changes, and Staged.Lock takes one. The service takes the locks of every document of a change set in the order of their keys, then calls Staged.Settle on each, which hashes each file again.

  • Every file still has the digest the stage read: the staged files are renamed as they are.
  • A file moved: another kapi process committed first, or a person saved. The home makes the stage's pass again under the locks, over the files as they now stand, with the editor starting afresh. An operation whose if_match the new content breaks is refused as stale with the content it found, and the change set is refused. Otherwise the new result is staged, the service runs its checks over that pass, and the files are hashed once more; a file that moved during that pass as well is refused as doc_changed.

Staged.Commit renames each staged file onto its target and marks it written, so a rename that fails after another landed leaves a partial result that names the files written. Staged.Release removes what was staged and drops the locks; the service releases them once the decisions and asset operations of the change set are applied and the change recorded. The service settles every document of a change set before it commits any, so a refusal found at settle time leaves every document as it was.

The lock is core/storage/filelock: flock(2) on Unix and LockFileEx on Windows, on a lock file of its own named for the file's resolved path, so two links to one file share it. The browser build has neither and runs one process, so there every lock on one path takes one mutex of the process, and two calls a page started at once take turns at the commit as two processes do natively. Inside a project the lock files sit under .kapi/work/locks/. The host passes Options.PrepareLocks, which the home runs once, before it first opens a lock file, to create the project's state directory with its ignore rule and to open the project's recorder, so a commit of the working tree leaves the lock files out, a recorder that cannot be opened stops the commit before anything is written, and a read, which takes no lock, leaves the project as it was. kapi apply, ksed -i, MCP, Kapi Desktop and the browser engine's kapiApply all edit a file of a project through the project's service, so they take one lock file for it. Outside a project the kapi host keeps them under locks in its data directory, so every kapi process of the user finds the same files; a caller that names no directory gets kapi-locks-<uid> in the temporary directory. The home creates the directory with mode 0700 and refuses one that others may write to, or a link, which another user could have made in advance to redirect the lock files. The lock orders kapi's processes on one machine. An editor saving between the second hash and the rename is a conflict the lock cannot see; the next read finds the change.

Options.BackupSuffix, when set, makes Staged.Commit copy each file it replaces to the file's path with the suffix appended, with the file's mode, before the rename. The copy is made under the lock once the file is known to hold what the stage read, so it holds the bytes the change was applied to; a file the change creates gets none. kapi apply --in-place=.bak and ksed -i.bak set it.

Options.BeforeSettle is called after a stage and before its first lock is taken. A test sets it to hold two writers at a barrier once both have staged against the same content. TestFileHome_PreparesTheLockDirectoryOnlyToCommit reads a document and finds no lock directory, then commits an edit and finds PrepareLocks called first.

Editions in files of their own​

In a project, the German edition of docs/guide.md is a file the recipe's target template names, read monolingually: its blocks hold German as their own content. A file the layout marks EditionFile.Bilingual (a PO catalog po/de.po beside the catalog po/en.po, a Qt Linguist file) is a translation catalog in its own right: it is read with the edition's language, its blocks hold German as their translation, and a unit with no translation holds no edition. When a change addresses an edition kept that way, the stage first reads the document once for each block's key and translation-invariant address, then reads the edition's file and pairs its blocks with the document's (a read that joins editions reads the document once, holding its blocks while it pairs them): by key, then by address (a heading written as its own identity, so a section pairs across languages), then by position when both files hold the same number of blocks. A monolingual file whose keys all sit under a root named for its language, where the document's sit under one named for the document's (a Rails catalog, en: in the document and de: in the German file), pairs by key below the root, so a translation one of them lacks leaves the rest paired. Each document block is given the edition from its partner while the editor sees it, and has it removed again before the document's own writer sees the block.

An edition the editor changed is written to its file:

  • When the file exists, it is read through its own skeleton and each partner's content replaced (its translation, in a bilingual file, written in the edition's language), so every other byte of the file stays: a comment, a header, or a section only that file has.
  • A translation created for a block that has no partner in a monolingual file arrives with a block of its own, which the format's writer adds to the staged bytes (restructureEditionFile, through format.StructureEditor.EditStructure, the call insert_block makes): under the key the file gives its keys (translateKey, so a file under de: takes the key under de:), after the partner of the document's nearest earlier block when that partner sits in the same mapping, else before the nearest later one on the same terms, else last in the mapping its key names. The staged bytes are read back, and a block that reads otherwise than it was given refuses the operation (verifyCreated). In a bilingual file, or a format whose writer adds no block (Markdown, HTML), such an edit is refused as unsupported, naming the block, because writing it would mean rewriting the file from the document's skeleton.
  • When the file does not exist yet, it is written from the document's skeleton with the writer set to the edition's locale, the blocks with no translation falling back to the document's own text, as kapi merge writes a target file. Its directory is created when the file is committed, so a preview, or a change set refused after the stage, leaves none.
  • A removal (remove_edition) from a monolingual file takes the partner block out of the file, since the block is the translation: the staged bytes are read back through the file's reader, each block is found there by its key (an edit beside the removal can make the reader skip an earlier block, a YAML value written as a number among them, which moves every block after it), and the format's writer removes it (restructureEditionFile, the call delete_block makes), with the result staged in their place. change.BaseOps lists remove_edition for a format that keeps one edition per file only where its writer declares delete_block (JSON, YAML, ARB). No file is deleted: a JSON translation whose last key is removed is written as {}.

A removal or a creation the writer cannot make (a YAML sequence item, named by its position; a key a value that reads as no block already holds; any block of a format whose writer adds none) refuses that operation alone, as unsupported naming the block and the file: the stage hands the refusal to the service's editor (change.EditionRefuser) and returns change.ErrRefused, so every other operation of the change set is not_applied, blocked by it, as a refusal in the pass leaves them. The read-backs of a bilingual file refuse a removal the same way.

Options.Materialize writes every edition file a change writes the second way, from the document's skeleton, whether or not the file exists: each block carries the edition the change gave it, else what its partner in the file held, else the document's own text. The file then follows the document's structure, so a block the document gained since the file was written takes its edition, and a block only the file held, or a comment, is gone. A monolingual file whose blocks have stopped pairing one to one with the document's is written this way even when the change set changes no content in it, and a write that gives the bytes the file holds is dropped, so the file is reported unwritten. A bilingual file that still pairs one to one keeps the first way, so its header (its language, its plural rule) and its comments stay; once the document gained or lost a unit it is written from the document's skeleton too, the document's header with it. kapi merge, its materializing form and kapi pull build their service with it, because each writes a whole translation where the other surfaces edit one in place.

An edition with no file yet, such as a held-back locale's drafts, is kept by a filehome.Keeper, the workspace home (core/workhome). The layout marks it EditionFile.Kept, with the file a delivery will write it to as its Path. The stage joins the kept edition (Keeper.Edition, each block's edition by the document's block key) to the document as it joins an edition file, and hands the editions the editor changed to the keeper as one filehome.KeptWrite instead of staging a file. The commit lock is the lock of the file the edition will be delivered to, so a delivery and a write to the kept edition take turns. Keeper.Commit stores the write only while the edition's head is still the token the stage read, and appends the change's record as it does, which the service hands it through change.RecordingStaged. A kept edition commits before any staged file is renamed, so a refusal there leaves every file as it was. A head that moved before the settle makes the stage's pass again, and one that moves after it refuses the change set as doc_changed. The result names the workspace home for that edition.

A reference to the edition's file itself ({"doc": "i18n/de/guide.md", "block": "installieren/p"}) resolves to the document and the edition, and the block key the German file reads with is translated to the document's through the same join (change.EditionKeyResolver). Results echo the canonical reference. A read of the German file shows the German edition as each block's own.

Adding and removing blocks​

A session is a change.StructuralSession and lists the structural operations its document's writer writes (format.StructuralOps), the writer configured for that document; filehome.Formats reports the list a built-in format's writer declares with its defaults. Describing a document, its blocks' operations in a read, and Apply follow the session's list. A stage whose Want is Structural begins with restructure (structure.go):

  1. It joins the editions the change set needs (every edition a removed block may hold, from DocInfo.Derived, and every edition a new block names) and reads the document, handing each block to the service's editor (change.Restructurer.Locate). The editor checks each operation against those blocks and returns the edits in change-set order.
  2. It reads the document's file and has the format's writer (format.StructureEditor.EditStructure) make every edit in it: a removed block by the block as its reader read it, a new block beside its anchor with the content of the document's own edition as its value.
  3. It does the same for each edition's file that exists: a removed block's partner in that file goes, and a new block that names the edition goes beside the partner of its anchor, or, with no anchor, last in the object or mapping its key names in that file. Where the file gives its keys a prefix of its own (de.nav.home where the document has en.nav.home), the new key takes that prefix. A new block whose anchor has no partner in the file is refused as unsupported for that edition; one whose anchor an earlier operation added goes beside that block when the earlier operation gave the file its edition too.
  4. It keeps the results in the session's overlay, and the rest of the stage reads them in place of the files: the join, the document's pass, and the writes of the editions. A file that does not exist yet is written in that pass from the document as the edits left it, the new block's edition given to the materialized file.

A writer's refusal (format.StructureError) refuses the operation it names: a key the document holds with no block (a number, a nested object) as invalid, anything else as unsupported. When the pass changes no content of a file the edits touched, the staged file is the writer's result itself; otherwise the pass's writer writes the result with the content edits in it. A structural edit holds the file whole in memory, and the streaming path applies to every other change. Settle makes the whole stage again, structural edits included, when a file moved.

A document a run produced​

A flow, a pull and a convergence pass's delivery write a document whole, and they commit it through the home too. Home.Produce(ctx, path, before, write) stages what write produces beside the file at path (atomicfile.StageWithParents), with the file's mode, hashing the bytes as they are written; before is the file's digest when the producer read it (filehome.Digest, empty for a file that did not exist). Produced.Commit opens the lock file the file's change sets lock (Home.lockPath, the same directory and the same PrepareLocks), takes it, and hashes the file again:

  • It holds before: the staged file is renamed onto it.
  • It holds the produced bytes already (the run changed nothing, or an identical run committed first): the staged file is discarded and nothing is written.
  • It holds anything else: the staged file is discarded, nothing is written, and Commit returns a *filehome.MovedError (errors.Is filehome.ErrMoved) naming the file and both digests.

The home holds the producer's bytes and not its operations, so a moved file is not applied again here. The kapi host follows each document of a flow run and applies the run's operations through the change service when the file moved, which reads the file again under the lock and refuses the operations whose if_match fails to hold. Produced.Release discards a staged file that was not committed. A home built with Options.WriteNothing stages nothing: Produce runs the producer into the digest alone and Commit writes nothing, which --print-ops uses. Options.WriteUnder names a directory such a home still writes in, as any home writes: a printing convergence pass's private draft tree, which its delivery gate reads. A home that only produces documents takes a nil layout; Open on it is refused.

A change's record (change.EditionChange.Block) is a copy of the block as the change left it, with its set of editions copied: the home takes the editions it joined from their own files out of the block once the writer has seen it, and the record still reads the stamp the change left on such an edition (TestFileHome_TheRecordReadsAnEditionInItsOwnFileAsTheChangeLeftIt).

Preview​

A preview stages each document and settles none: the service releases every stage after the commit check, so the temporary files are removed and the documents keep their bytes. Staged.Diff renders a unified diff of each file it would write, from the staged bytes against the file, when both are text of at most 1 MiB; an archive member is diffed as the member.

Tests​

  • changetest.Run is the conformance suite, run by TestFileHome_Conformance on two JSON documents with mode 0640, a French PO catalog whose translation it removes and creates again, and a JSON document whose French it removes from fr/d.json and creates again there; its interleaving cases run a second sender from Options.BeforeSettle. TestFileHome_RemovesATranslationFromItsOwnFile removes a nested key's translation from de/guide.json in place and under Options.Materialize, TestFileHome_RemovingEveryTranslationKeepsTheFile leaves {}, and TestFileHome_RefusesARemovalTheFormatCannotWrite refuses one from a Markdown translation and writes nothing. TestFileHome_ARemovalBesideAnEditTakesOutTheBlockItNames removes a YAML translation beside an edit that makes an earlier block read as none, in place and materialized; TestFileHome_AWriterRefusalRefusesTheRemovalAlone refuses a sequence item's removal alone and beside an edit, which is not_applied; TestFileHome_ARemovedTranslationIsCreatedAgain removes a translation from JSON, YAML, a YAML file under its language's root and ARB, checks every other translation still pairs, and creates it again beside the block it sat by; TestFileHome_ATranslationIsCreatedInAFileWithNoBlockLeft and TestFileHome_ACreationTheWriterCannotMakeIsRefused cover an empty file and a key a number holds. The removal cells of the operations matrix (core/formats/opsmatrix_removal_test.go) hold JSON, YAML and ARB to their bytes, a translation created again included, a YAML sequence item to the refusal, and every other format with one edition per file to the refusal. TestFileHome_RemovingATranslationABilingualFileHolds removes a translation from a PO catalog and refuses the removal from XLIFF 1.2, XLIFF 2 and TMX, writing nothing; TestFileHome_RemovingATranslationFromItsOwnBilingualFile does the same where the translation is a catalog of its own, and leaves the document's file as it was; TestFileHome_ABilingualRemovalRefusalHoldsTheEditBesideIt refuses the XLIFF removal alone, the edit beside it not_applied.
  • TestFileHome_TwoProcessesEditingDifferentBlocksLoseNoEdit starts the test binary as two writer processes over 50 rounds; each stages an edit of a different paragraph of one HTML file and waits at a barrier until both have staged. A third of the rounds let the first writer finish before the second settles, a third the reverse, and a third let the lock decide. Both edits are in the file after every round. Settling without the second pass loses an edit in the first round. It skips under GOOS=js, which cannot start a process.
  • TestFileHome_TwoWritersOfOneBlockConflict stages two edits of one block against one content in one process: one lands and the other is refused as stale with what the first wrote. make test-wasm-stores runs it under GOOS=js too, where only the lock's mutex orders the two.
  • TestFileHome_ALargeEditStaysInBoundedMemory edits one message of a 100,000-message JSON catalog of about 8 MiB and asserts the heap the edit takes stays under 24 MiB; the streaming path takes under 2 MiB, and a buffered round trip of the same catalog takes about 80 MiB.
  • TestFileHome_ADocumentThatMovedOnceIsAppliedAgain and TestFileHome_ADocumentThatKeepsMovingIsDocChanged move the file between the stage and the commit.
  • The edition tests write a translation through its own file, materialize a missing one without creating its directory before the commit, refuse an edit that would rewrite an existing Markdown translation (alone, with an edit beside it not_applied), name the editions a source edit leaves stale, and edit an archive member in place. TestFileHome_MaterializeWritesAnEditionFileFromTheDocument writes a block an existing translation lacks under Options.Materialize, keeping the translation's other messages in the document's order and dropping one only the translation held, where the same change without it adds the key beside the one before it and keeps the rest. TestFileHome_MaterializeBringsADriftedEditionFileBackToTheDocument writes a translation whose blocks have stopped pairing with the document's under a change that changes no content, and leaves one that still pairs unwritten. TestFileHome_ABilingualTranslationFileHoldsTheEditionAsItsTranslation reads a French PO catalog beside an English one as its msgstr, and keeps every translation the change leaves and the catalog's header in place, under Options.Materialize too, until the source gains a message.
  • TestFileHome_AnEditionFileThatMovesDuringTheStageIsReadAgain lets another writer save the German file right after the stage read it: the edit is applied again to what that writer wrote, and an edit of the same block is refused as stale.
  • TestFileHome_KeepsABackupOfWhatACommitReplaces writes a backup with the original bytes and mode on a commit, and none for a preview, a refusal or an unchanged edit; TestService_ReadEachReadsTheWholeDocument reads past the most a page holds in one pass.
  • The structure tests add and remove keys of JSON, YAML and ARB catalogs with content edits beside them, write a new block's edition into an existing translation and into one written for the first time, remove a block from every translation, take a file's key prefix, put a new key with no anchor in the mapping its key names in every file, make the edits again when the file moved, and refuse a key held without a block, an anchor with no partner in a translation, and a format that adds no blocks. The operations matrix (core/formats/opsmatrix_structural_test.go) holds each format that declares the operations to its cells.
  • TestFileHome_OneFileNamedTwiceIsRefused names two members of one archive, and a file and a link to it, in one change set; TestFileHome_ACommitInterruptedAfterOneFileReportsWhatLanded stops a rename after another landed; TestFileHome_ABilingualFileTakesOnlyTheEditionsItHolds edits the translation an XLIFF file and a PO catalog hold and refuses the editions they cannot hold; TestFileHome_KeepsItsLockDirectoryToItsUser refuses a lock directory others may write to; TestFileHome_PreparesTheLockDirectoryAtTheFirstCommit runs PrepareLocks once, at the first commit and never on a read.
  • TestProduce_CommitsWhatTheRunWrote commits a produced document over the file it read, creates a new file's directory only at the commit, leaves a file that already holds the bytes, and keeps a file a person saved or another writer created meanwhile, reporting it moved; TestProduce_TakesTheLockAChangeSetCommitsUnder holds a change set's commit lock and finds the produced document waiting for it, then refused because the change set landed first; TestProduce_AHomeThatWritesNothingStagesNothing, TestProduce_AHomeThatWritesNothingStillWritesUnderItsTree and TestProduce_KeepsTheModeOfTheFileItReplaces cover the rest.
  • In core/workhome, TestWorkspaceHome_Conformance runs the conformance suite on the German edition of a JSON file that no file holds, TestWorkspaceHome_KeepsTheEditionInTheLog writes a kept edition that lands in no file, is recorded once and reads back with the status and origin the write left, and TestWorkspaceHome_RefusesAStaleHead refuses as doc_changed a commit staged on a head another writer moved, storing nothing.
  • In the host, TestChangeService_AProjectCatalogsTranslationLivesInItsTargetFile reads and writes the French of a PO catalog and of a PO template in po/fr.po, with and without a target language and through either file, leaving the source's bytes; the host and Kapi Desktop conformance runs remove the French from po/fr.po beside po/en.po, and the host's from locales/fr.json beside locales/en.json too. TestChangeService_ASourceDocumentResolvesWithoutTheWholeRecipe locates a source file and its translations without expanding a pattern, TestChangeService_AReadWritesNothing reads a project without creating its state directory, and TestMCPApplyEdits_EachProjectInItsOwnSourceLanguage edits a translation in a project written in the language another project translates into.