Çöñţéñţ ḿéḿöŕý

neokapi'š çöñţéñţ ḿéḿöŕý ļîṽéš îñ ţĥé memory/ þàçķàĝé. Îţ ŵöŕķš ŵîţĥ ţĥé ƒüļļ çöñţéñţ ḿöđéļ: éàçĥ éñţŕý ĥöļđš ḿüļţîļîñĝüàļ ṽàŕîàñţš àš Run šéǫüéñçéš (ţéẋţ þļüš îñļîñé ḿàŕķüþ) àñđ îš ḿàţçĥéđ îñ ţĥŕéé ţîéŕš ŵîţĥ éñţîţý-àŵàŕé àđàþţàţîöñ. Ţĥé šàḿé éñĝîñé ƃàçķš ţĥé kapi memory çöḿḿàñđš, ţĥé recycle þîþéļîñé ţööļ, ţĥé ţŕàñšļàţé šţéþ'š þŕîöŕ-ṽéŕšîöñ ŕéüšé, àñđ ţĥé Ĝö ļîƃŕàŕý.

Îñ ţĥé ÇĻÎ, çöñţéñţ ḿéḿöŕý îš ţĥé éñĝîñé üñđéŕ kapi exec recycle (ţĥé šîñĝļé-ţööļ ļéṽéŕàĝé þàšš) àñđ üñđéŕ ţĥé ƒîŕšţ šţéþ öƒ kapi up'š đéƒàüļţ ƒļöŵ, ŵĥîçĥ ŕéçýçļéš ƒŕöḿ ḿéḿöŕý ƃéƒöŕé àñý ÀÎ ţŕàñšļàţîöñ ŕüñš. Šéé Üñđéŕšţàñđîñĝ ţĥé ÇĻÎ ļàýéŕš ƒöŕ ĥöŵ ţĥé šîñĝļé-ţööļ, ƒļöŵ, àñđ þŕöĵéçţ-ļööþ šüŕƒàçéš ŕéļàţé.

Çöñţéñţ-àŵàŕé ḿàţçĥîñĝ

Éàçĥ éñţŕý îš îñđéẋéđ üñđéŕ ţĥŕéé ķéýš, ţŕîéđ îñ öŕđéŕ, šö ţĥé ĥîĝĥéšţ-ǫüàļîţý ḿàţçĥ îš ŕéţüŕñéđ ƒîŕšţ:

Ţîéŕ	Ḿàţçĥ ţýþé	Ñöŕḿàļîžéš	Éẋàḿþļé
1	Ĝéñéŕàļîžéđ	Ñàḿéđ éñţîţîéš → ţýþéđ þļàçéĥöļđéŕš	"Ŵéļçöḿé, Ĵöĥñ" → "Ŵéļçöḿé, {PERSON}"
2	Šţŕüçţüŕàļ	Îñļîñé ḿàŕķüþ → ñöŕḿàļîžéđ çöđéš	"Çļîçķ ĥéŕé" → "Çļîçķ {1}ĥéŕé{/1}"
3	Þļàîñ	Ñöţĥîñĝ (ŕàŵ ţéẋţ)	Ļéṽéñšĥţéîñ ƒüžžý ḿàţçĥîñĝ

Éàçĥ ţîéŕ ýîéļđš éẋàçţ (100%) öŕ ƒüžžý ḿàţçĥéš. Ŵĥéñ à ĝéñéŕàļîžéđ éẋàçţ ḿàţçĥ îš ƒöüñđ, éñţîţý ṽàļüéš ƒŕöḿ ţĥé çüŕŕéñţ šöüŕçé àŕé àđàþţéđ îñţö ţĥé šţöŕéđ ţàŕĝéţ, šö "Ŵéļçöḿé, Ƃöƃ" → "Ƃîéñṽéñüé, Ƃöƃ" àđàþţš ţö "Ŵéļçöḿé, Àļîçé" → "Ƃîéñṽéñüé, Àļîçé" àţ 100%. Ţĥîš öŕđéŕîñĝ ḿîŕŕöŕš ĥöŵ à ţŕàñšļàţöŕ éṽàļüàţéš ḿàţçĥéš: éñţîţý đîƒƒéŕéñçéš ḿàţţéŕ ļéšš ţĥàñ šţŕüçţüŕàļ öñéš, ŵĥîçĥ ḿàţţéŕ ļéšš ţĥàñ ţéẋţüàļ çĥàñĝéš.

Ţĥé ţýþéđ þļàçéĥöļđéŕš ţĥé ĝéñéŕàļîžéđ ţîéŕ ķéýš öñ ({PERSON}, {PRODUCT}, …) çöḿé ƒŕöḿ éñţîţý đéţéçţîöñ, à ƒàšţ ļöçàļ ḿöđéļ öŕ àñ ĻĻḾ ţĥàţ ŕéçöĝñîžéš ţĥé ñàḿéđ ţĥîñĝš îñ à ƃļöçķ. Ýöü đöñ'ţ ŕüñ đéţéçţîöñ àš à šéþàŕàţé ţàšķ: îţ ĥàþþéñš àš þàŕţ öƒ þŕéþàŕîñĝ çöñţéñţ, àñđ ţĥé šàḿé đéţéçţîöñ àļšö þöŵéŕš ŕéđàçţîöñ. Àññöţàţé éñţîţîéš öñçé àñđ ƃöţĥ ĝéñéŕàļîžéđ ḿéḿöŕý ŕéüšé àñđ ŕéđàçţîöñ ƒöļļöŵ.

Šţöŕàĝé ƃàçķéñđš

Ţŵö ƃàçķéñđš šĥîþ îñ ţĥé memory/ þàçķàĝé, ƃöţĥ îḿþļéḿéñţîñĝ ţĥé ContentMemory îñţéŕƒàçé ŵîţĥ ƒüļļ ţîéŕ šüþþöŕţ:

Îñ-ḿéḿöŕý (memory.NewInMemoryStore): ƒàšţ àñđ éþĥéḿéŕàļ, üšéđ ƒöŕ šéššîöñ-šçöþéđ ƃàţçĥ þŕöçéššîñĝ.
ŠǪĻîţé (memory.NewSQLiteStore): þéŕšîšţéñţ ƒîļé-ƃàšéđ šţöŕàĝé ƒöŕ ÇĻÎ ŵöŕķƒļöŵš.

Ţĥé îñţéŕƒàçé àļšö àççöḿḿöđàţéš šéŕṽéŕ-šîđé ƃàçķéñđš ƒöŕ ḿüļţî-üšéŕ đéþļöýḿéñţš ŵîţĥ þŕöĵéçţ šçöþîñĝ, šţŕéàḿš, àñđ ŵöŕķšþàçé îšöļàţîöñ. Ƒüžžý ḿàţçĥîñĝ üšéš Ļéṽéñšĥţéîñ éđîţ đîšţàñçé ŵîţĥ à çöñƒîĝüŕàƃļé ţĥŕéšĥöļđ (đéƒàüļţ 0.70); ŕéšüļţš àŕé šöŕţéđ ƃý šçöŕé àñđ ţĥéñ ƃý ţîéŕ.

ÇĻÎ üšàĝé

Ŕéšöüŕçé ļöçàţîöñ

Àļļ kapi memory çöḿḿàñđš (éẋçéþţ list) àççéþţ ţĥéšé ḿüţüàļļý éẋçļüšîṽé ƒļàĝš:

Ƒļàĝ	Ŕéšöļṽéš ţö	Éẋàḿþļé
`--name <n>`	`~/.config/kapi/memory/<n>.db`	`--name project-memory`
`--local`	`./memory.db` (çüŕŕéñţ đîŕéçţöŕý)	`--local`
`--file <path>`	Éẋþļîçîţ ƒîļé þàţĥ	`--file /shared/memory.db`
(ñö ƒļàĝ)	Šàḿé àš `--local`

Đàţàƃàšéš àŕé çŕéàţéđ öñ đéḿàñđ îƒ ţĥéý đöñ'ţ éẋîšţ. memory/ àñđ memory.db àŕé ţĥé öñ-đîšķ ñàḿéš ƒöŕ à ñàḿéđ öŕ ļööšé çöñţéñţ ḿéḿöŕý. ~/.config/kapi îš ţĥé üšéŕ çöñƒîĝ đîŕéçţöŕý öñ Ļîñüẋ, àñđ ŕéšöļṽéš ţö ~/Library/Application Support/kapi öñ ḿàçÖŠ. kapi config path þŕîñţš ţĥé ŕéšöļṽéđ ļöçàţîöñ.

Ŵîţĥ ñö ƒļàĝ îñšîđé à þŕöĵéçţ, ţĥé çöñţéñţ ḿéḿöŕý îš îñšţéàđ à šéţ öƒ ţàƃļéš îñ ţĥé þŕöĵéçţ'š .kapi/work/store.db; šéé Ḿéḿöŕý & ţéŕḿš šţöŕàĝé.

kapi memory import translations.memory.json --name project-memory
kapi memory export --name project-memory -o project.memory.json
kapi memory lookup "Welcome to our platform" --name project-memory -s en -t fr
kapi memory search "welcome" --name project-memory -s en
kapi memory stats --name project-memory
kapi memory list

Þîþéļîñé îñţéĝŕàţîöñ

Ţĥé recycle ţööļ ǫüéŕîéš çöñţéñţ ḿéḿöŕý ƒöŕ éàçĥ Ƃļöçķ'š šöüŕçé šéĝḿéñţš àñđ àþþļîéš ḿàţçĥéš. Éṽéŕý ḿàţçĥ, éẋàçţ öŕ ƒüžžý, îš ŕéçöŕđéđ àš àñ AltTranslation àññöţàţîöñ (ḿàţçĥéđ šöüŕçé/ţàŕĝéţ ŕüñš, šçöŕé, ḿàţçĥ ţýþé), àñđ à ƒîļļéđ ţàŕĝéţ îš çöḿḿîţţéđ ŵîţĥ þŕöṽéñàñçé (Origin{Kind: "memory", Tool: "recycle"}), îţš šçöŕé, àñđ draft šţàţüš, šö ţĥé ļéṽéŕàĝé îš àüđîţàƃļé ŕàţĥéŕ ţĥàñ àñ öþàǫüé öṽéŕŵŕîţé. Éẋàçţ ḿàţçĥéš šķîþ ÀÎ ţŕàñšļàţîöñ, ŕéđüçîñĝ çöšţ àñđ ļàţéñçý.

Šéĝḿéñţ-àŵàŕé ļéṽéŕàĝé. Ŵĥéñ à ƃļöçķ çàŕŕîéš à ḿüļţî-šéĝḿéñţ šéĝḿéñţàţîöñ öṽéŕļàý (à þŕöšé þàŕàĝŕàþĥ šþļîţ îñţö šéñţéñçéš), recycle ļööķš üþ çöñţéñţ ḿéḿöŕý þéŕ šéñţéñçé. Ţĥîš ŕéçöṽéŕš ļéṽéŕàĝé ƒöŕ ḿüļţî-šéñţéñçé ƃļöçķš ţĥàţ ŵöüļđ ñéṽéŕ ḿàţçĥ ţĥé šéñţéñçé-ķéýéđ ḿéḿöŕý àš à šîñĝļé üñîţ. À šîñĝļé-šéĝḿéñţ ƃļöçķ (ḿöšţ ÜÎ šţŕîñĝš) ţàķéš ţĥé ŵĥöļé-ƃļöçķ þàţĥ üñçĥàñĝéđ.

Ţĥé ŕéšüļţ îš ŕéçöŕđéđ šö îţ îš àüđîţàƃļé:

Éàçĥ ḿàţçĥîñĝ šéñţéñçé îš àţţàçĥéđ àš àñ AltTranslation àññöţàţîöñ (ḿàţçĥéđ šöüŕçé àñđ ţàŕĝéţ ŕüñš, šçöŕé, éẋàçţ/ƒüžžý ḿàţçĥ ţýþé), ķéþţ ŵĥéţĥéŕ öŕ ñöţ ţĥé ƃļöçķ ţàŕĝéţ îš ƒîļļéđ, šö þàŕţîàļ ļéṽéŕàĝé (šöḿé šéñţéñçéš ḿàţçĥéđ, šöḿé ñéŵ) îš þŕéšéŕṽéđ ƒöŕ à ŕéṽîéŵéŕ öŕ à ļàţéŕ ţŕàñšļàţîöñ šţàĝé ŕàţĥéŕ ţĥàñ đîšçàŕđéđ.
Ţĥé ƃļöçķ ŕéçöŕđš tm-segment-matches (é.ĝ. 3/5) ƒöŕ ǫüîçķ ĝàţîñĝ.
Ţĥé ƃļöçķ ţàŕĝéţ îš ƒîļļéđ öñļý ŵĥéñ éṽéŕý šéñţéñçé ḿàţçĥéđ àñđ ţĥé šéĝḿéñţš àŕé çöñţîĝüöüš; ŵĥéñ îţ îš, ţĥé çöḿḿîţţéđ ţàŕĝéţ çàŕŕîéš þŕöṽéñàñçé (Origin{Kind: "memory", Tool: "recycle"}), ţĥé ŕöļļ-üþ Score, àñđ draft šţàţüš: à ŕéṽîéŵàƃļé þŕé-ƒîļļ ŕàţĥéŕ ţĥàñ à šîĝñéđ-öƒƒ ţŕàñšļàţîöñ.

Ŕüñ šéĝḿéñţàţîöñ ƃéƒöŕé recycle ţö éñàƃļé ţĥîš.

kapi exec recycle input.html -o output.html --source-lang en --target-lang fr --memory project-memory

steps:
  - tool: recycle
    config:
      fuzzyThreshold: 70 # minimum score for fuzzy matches (0-100)
      fillTarget: true # copy the best candidate into the target
      fillTargetThreshold: 95 # minimum score required to fill the target

Ţĥé ţŵö ţĥŕéšĥöļđš đéƒàüļţ ţö 70 àñđ 95. Ţĥé ƒüžžý þàţĥ ƃéļöŵ àñ éẋàçţ ḿàţçĥ îš ŕéţîŕîñĝ îñ ƒàṽöüŕ öƒ ţĥé ĝöṽéŕñéđ ŕéüšé ƃéļöŵ, àñđ ţĥé đéƒàüļţš šţàý ŵĥéŕé ţĥéý àŕé öñ ţĥé þàţĥš ţĥàţ ŕüñ ŵîţĥöüţ à ţŕàñšļàţé šţéþ.

Ĝöṽéŕñéđ þŕîöŕ-ṽéŕšîöñ ŕéüšé

Ŕéçýçļé àñšŵéŕš ƃý šöüŕçé ţéẋţ: ţĥé šàḿé šéñţéñçé, öŕ öñé çļöšé ţö îţ, ĝéţš ţĥé šàḿé ţŕàñšļàţîöñ. Ţĥé ţŕàñšļàţé šţéþ àšķš à šéçöñđ ǫüéšţîöñ ƃý üñîţ: ŵĥàţ ŵàš ţĥé þŕéṽîöüšļý àþþŕöṽéđ ţŕàñšļàţîöñ öƒ ţĥîš ƃļöçķ, ƃéƒöŕé îţš šöüŕçé ŵàš éđîţéđ? Ţĥé àñšŵéŕ ĝöéš îñţö ţĥé þŕöḿþţ àš ŕéƒéŕéñçé, šö ţĥé ḿöđéļ ŕéđŕàƒţš ƒŕöḿ ŵöŕđîñĝ à þéŕšöñ àļŕéàđý àççéþţéđ ŕàţĥéŕ ţĥàñ ƒŕöḿ ñöţĥîñĝ.

Ţĥé ŕéàđ îš ĝàţéđ öñ ĝöṽéŕñàñçé. À VersionRequest ñàḿéš ţĥé üñîţ (ţĥé ƃļöçķ'š îđéñţîţý àçŕöšš éđîţš), ţĥé þöîñţ ţĥé çöñţéñţ šîţš àţ, ţĥé šöüŕçé àñđ ţàŕĝéţ ļöçàļéš, àñđ GovernedBy, ţĥé ƒîñĝéŕþŕîñţ öƒ ţĥé ṽöîçé ĝüîđàñçé àñđ ţéŕḿ ŕüļéš àƃöüţ ţö ŕéàçĥ ţĥé ḿöđéļ. Àñ àñšŵéŕ àþþŕöṽéđ üñđéŕ àñý öţĥéŕ çöñţéẋţ îš ŵîţĥĥéļđ, ƃéçàüšé à ţŕàñšļàţîöñ àççéþţéđ üñđéŕ à đîƒƒéŕéñţ ṽöîçé öŕ ṽöçàƃüļàŕý îš à ŵŕöñĝ ŕéƒéŕéñçé ŕàţĥéŕ ţĥàñ à ĥéļþƒüļ öñé. Ţĥé ţŕàñšļàţé ţööļ'š reuse šéţţîñĝ (prior, ţĥé đéƒàüļţ, öŕ none) ţüŕñš ţĥé ŕéàđ öñ öŕ öƒƒ; ţĥé context: šéţţîñĝ îš šéþàŕàţé, ƃéçàüšé à þŕîöŕ ṽéŕšîöñ çöšţš à çöŕþüš ŕéàđ ŵĥéŕé à ƃļöçķ'š ñéîĝĥƃöüŕš çöḿé ƒŕéé ƒŕöḿ ţĥé đöçüḿéñţ îñ ĥàñđ.

v, ok := corpus.PriorVersion(ctx, memory.VersionRequest{
    Unit:       block.ChainUnit(),
    Point:      "kapimart/web",
    Source:     "en",
    Target:     "nb",
    GovernedBy: contextFingerprint,
})

Ĝö ļîƃŕàŕý

Îñţéŕƒàçé

type ContentMemory interface {
    Add(entry Entry) error
    Lookup(source *model.Block, sourceLocale, targetLocale model.LocaleID,
        opts LookupOptions) ([]Match, error)
    LookupText(source string, sourceLocale, targetLocale model.LocaleID,
        opts LookupOptions) ([]Match, error)
    Delete(id string) error
    Count() int
    Close() error
}

(Ḿéţĥöđš, àñđ ţĥé ŢḾẊ ĥéļþéŕš ƃéļöŵ, ţàķé à context.Context àš ţĥéîŕ ƒîŕšţ àŕĝüḿéñţ îñ ţĥé ŕéàļ ÀÞÎ; îţ îš éļîđéđ ĥéŕé ƒöŕ ŕéàđàƃîļîţý.)

Lookup ţàķéš à ƒüļļ *model.Block àñđ üšéš îţš Ŕüñ çöñţéñţ (àñđ éñţîţý àññöţàţîöñš) ƒöŕ ţîéŕéđ ḿàţçĥîñĝ; LookupText ţàķéš à þļàîñ šţŕîñĝ àñđ þéŕƒöŕḿš þļàîñ-ţîéŕ ḿàţçĥîñĝ öñļý. LookupSegment ḿàţçĥéš à šîñĝļé šéĝḿéñţ öƒ à ƃļöçķ ƒöŕ šéñţéñçé-ļéṽéļ ļéṽéŕàĝé. Ƃöţĥ ŠǪĻîţé àñđ îñ-ḿéḿöŕý ƃàçķéñđš àļšö îḿþļéḿéñţ EntryProvider (Entries()), ŵĥîçĥ îš ĥöŵ ŢḾẊ éẋþöŕţ éñüḿéŕàţéš à šţöŕé, àñđ öƒƒéŕ þàĝîñàţéđ SearchEntries(...) ƒöŕ ƃŕöŵšîñĝ.

Ķéý ţýþéš

type Entry struct {
    ID          string
    ProjectID   string
    Variants    map[model.LocaleID][]model.Run // peer language variants
    HintSrcLang model.LocaleID                 // locale the author treated as canonical
    Entities    []EntityMapping                // entity placeholders
    Properties  map[string]string
    Origins     []Origin
    Note        string
    CreatedAt   time.Time
    UpdatedAt   time.Time
}

type Match struct {
    Entry             Entry
    Score             float64 // 0.0-1.0
    MatchType         MatchType
    ProjectID         string             // provenance of the matched entry
    EntityAdaptations []EntityAdaptation // entity value substitutions
}

type LookupOptions struct {
    MinScore     float64      // minimum match score (default 0.7)
    MaxResults   int          // max results to return (default 10)
    MatchModes   []MatchMode  // which tiers to use (default: all)
    ProjectID    string       // project context for scoring boost
    ProjectScope ProjectScope // project filtering mode (default: all)
}

Àñ éñţŕý îš ḿüļţîļîñĝüàļ: ţĥéŕé îš ñö àüţĥöŕîţàţîṽé šöüŕçé àţ ţĥé þéŕšîšţéñçé ļàýéŕ; éàçĥ ļàñĝüàĝé îš à þééŕ Variants[locale] Ŕüñ šéǫüéñçé, àñđ ţĥé ļööķüþ đîŕéçţîöñ îš šüþþļîéđ àţ ţĥé çàļļ šîţé. MatchType ŕàñĝéš ƒŕöḿ generalized-exact (ĥîĝĥéšţ ŕéüšé) ţĥŕöüĝĥ structural-exact, exact, ţĥé çöŕŕéšþöñđîñĝ ƒüžžý ṽàŕîàñţš, đöŵñ ţö fuzzy. Entry ĥéļþéŕš: Variant(locale), VariantText(locale), VariantStructural(locale), VariantGeneralized(locale). Ţĥé EntityAdaptations ƒîéļđ öñ à ḿàţçĥ ļîšţš éàçĥ šüƃšţîţüţîöñ ŵîţĥ îţš þöšîţîöñ šö çöñšüḿéŕš çàñ àþþļý àđàþţàţîöñš þŕéçîšéļý.

Éẋàḿþļé

package main

import (
    "fmt"

    "github.com/neokapi/neokapi/core/model"
    "github.com/neokapi/neokapi/memory"
)

func main() {
    tm := memory.NewInMemoryStore()
    defer tm.Close()

    tm.Add(memory.Entry{
        ID: "e1",
        Variants: map[model.LocaleID][]model.Run{
            "en": {{Text: &model.TextRun{Text: "Welcome to our platform"}}},
            "fr": {{Text: &model.TextRun{Text: "Bienvenue sur notre plateforme"}}},
        },
        HintSrcLang: "en",
    })

    block := model.NewBlock("b1", "Welcome to our platform")
    matches, err := tm.Lookup(block, "en", "fr", memory.DefaultLookupOptions())
    if err != nil {
        panic(err)
    }
    for _, m := range matches {
        fmt.Printf("Score: %.0f%% Type: %s Target: %s\n",
            m.Score*100, m.MatchType, m.Entry.VariantText("fr"))
    }
}

ŢḾẊ îḿþöŕţ / éẋþöŕţ

// Importing keeps only the named bilingual pair; ImportTMXLocalePairs keeps an
// arbitrary set of locales, and an empty set keeps every <tuv> present.
count, err := memory.ImportTMXWithOptions(tm, reader, "en", "fr",
    memory.ImportTMXOptions{OriginKey: "corpus.tmx"})

// ExportTMX emits one <tu> per entry with a <tuv> per selected locale; an
// empty locale list keeps every variant present.
err = memory.ExportTMX(tm, writer, []model.LocaleID{"en", "fr"})

Îḿþöŕţ ŕéǫüîŕéš à ƃàçķéñđ ţĥàţ šüþþöŕţš îḿþöŕţ šéššîöñš, ţĥàţ îš, öñé îḿþļéḿéñţîñĝ ţĥé þéŕšîšţéñţ Store îñţéŕƒàçé.