Skip to main content

AD-011: AI Providers

Summary​

The framework integrates LLM capabilities through an LLMProvider interface in providers/ai/ (package aiprovider), with built-in implementations for Anthropic, OpenAI, Azure OpenAI, Ollama, and Gemini (plus an offline demo provider) and an optional StreamingLLMProvider extension for live thinking progress. ollama is the built-in on-device backend (driving a local Ollama runtime over HTTP). Plugins can register further providers at runtime, so the live provider set is whatever aiprovider.Providers() returns, not a fixed list. Messages are multimodal — content is an ordered list of text, image, audio, or video parts — and each provider advertises the input modalities it accepts, so a vision- or audio-capable model reads a Block's media anchor (AD-002) directly. AI tools call providers directly; throughput comes from config-driven batching and bounded concurrency inside the tool, not from a separate worker-pool subsystem. A ChatStructured method with JSON Schema enables reliable batch translation and other structured-output tasks.

Context​

Modern LLMs are capable translators, reviewers, and terminology extractors. Treating them as a separate service loses the composability of the streaming pipeline: AI tools should sit alongside memory leverage, term enforcement, and QA in the same flow.

AI APIs come with practical constraints: rate limits, cost per token, transient failures, and variable latency. The framework's answer is to keep the provider interface thin and let the calling tool decide how much work to batch into a single request and how many requests to run in parallel. Workspace-scale orchestration — async job queues, multi-tenant quotas — belongs to a platform layer, not to the framework primitives.

Providers also differ in their structured-output mechanism: OpenAI and Azure OpenAI use response_format: json_schema, Anthropic uses tool-use with input_schema, Ollama uses format: json, and Gemini uses response schema hints. A single interface must paper over these details while giving tools a predictable contract.

Decision​

LLMProvider interface​

type LLMProvider interface {
    Name() ProviderID
    InputModalities() []Modality // non-text inputs accepted; text always
    Translate(ctx context.Context, req TranslateRequest) (*TranslateResponse, error)
    Chat(ctx context.Context, messages []Message) (*ChatResponse, error)
    ChatStructured(ctx context.Context, messages []Message,
        schema JSONSchema) (*ChatResponse, error)
    Close() error
}

ChatStructured extends Chat with a JSON Schema constraint that forces the provider to return structured output. The JSONSchema type includes Name, Description, Schema (the JSON Schema definition), and a Strict flag for providers that support strict validation.

Provider configuration is schema-driven: fields in AI tool configs generate CLI flags automatically via schema.FromStruct(), removing the need for manual flag registration.

Multimodal content​

A Message's content is an ordered list of typed parts, so one interface carries text, images, audio, and video:

type Message struct {
    Role  string        // "system" | "user" | "assistant"
    Parts []ContentPart
}

type ContentPart struct {
    Kind  ContentKind  // closed set, named type
    Text  string       // Kind == ContentText
    Media *model.Media // otherwise — a bounded slice, carried by reference
}

type ContentKind string

const (
    ContentText  ContentKind = "text"
    ContentImage ContentKind = "image"
    ContentAudio ContentKind = "audio"
    ContentVideo ContentKind = "video"
)

A media part carries its payload as a model.Media — the framework's binary-reference type (AD-002), with precedence BlobKey > URI > Data — not a bare []byte. A small slice rides inline (Data); a larger one (a video clip) is a BlobKey/URI and is never forced into memory. A single helper at the provider's HTTP boundary resolves the Media to the backend's wire form (base64 inline, or a fetchable URL where the provider supports it), so provider implementations stay storage-agnostic — they never read a file or the blob store. This keeps one binary idiom across Media, plugin I/O, and provider content.

A text-only message is a single text part, so translation, QA, and terminology tools use the interface with no media parts — the common path carries no media ceremony. Image, audio, and video parts carry a Block's media slice (AD-002) into the prompt, which is what the multimodal extraction refinement tier sends (AD-030).

Backends differ in which input modalities they accept, so InputModalities() advertises a provider's reach (Modality being the image/audio/video subset of ContentKind; text is always accepted) and a caller selects a provider that fits rather than discovering the limit at call time:

ProviderAccepts
Geminitext, image, audio, video
OpenAI / Azure OpenAItext, image (audio on audio-capable models)
Anthropictext, image
Ollama (vision models)text, image

Built-in providers​

ProviderFileNotes
Anthropicproviders/ai/anthropic.goExtended thinking support
OpenAIproviders/ai/openai.goresponse_format JSON schema
Azure OpenAIproviders/ai/azureopenai.goManaged Identity via TokenProvider
Ollamaproviders/ai/ollama.goOn-device local models (GPU); no key. Streaming, format: json, options + keep_alive, reasoning disabled. Managed via kapi models ollama
Google Geminiproviders/ai/gemini.goSSE streaming with includeThoughts

Default models are deliberately absent from this table — they change with every model generation, and a hardcoded list here would be wrong within weeks. They live in the model catalog (below), and the current set is on the generated AI Models reference.

Two non-network providers round out the registry: a mock provider (providers/ai/mock.go) for deterministic tests, and a demo provider (providers/ai/demo.go) registered as demo that returns illustrative output so the browser playground can run AI commands with no API keys. The provider list is generated from the registry in providers/ai/provider.go (Providers()), not hardcoded — the live set surfaces as the provider option in the translate reference.

The model catalog​

The models kapi supports are described in one place: providers/ai/models.json, a curated catalog embedded into the framework and read as aiprovider.Models(). It is the single source of truth, and the rest derives from it.

Why data, and why curated. One catalog rather than model knowledge spread over a DefaultXModel constant per provider, a prefix→ceilings map, and a price table under scripts/batcheval — none of which answers the question a user actually asks: is this model current, superseded, or retired, and since when? The catalog carries the defaults and the ceilings (LimitsForModel resolves through it; a test asserts every provider default is catalogued and marked default_for that provider) as well as the lifecycle. It is data because a model list hardcoded in Go goes stale silently; it is curated because the vendors' APIs return only what is live today as a flat list of ids — they do not say when a model entered neokapi, what replaced it, or when it retires. Those are facts about our support, and only a human (or an agent reading a model card) can supply them.

Each entry carries the model's provider, aliases, output/context ceilings, and its lifecycle: status (active | superseded), introduced, superseded_by, and retirement_date. There is no retired status: a model the provider stops serving is removed from the catalog, not kept as a tombstone — the catalog is the list of models kapi supports, and a dead model supports nothing. An announced future retirement is a date on a still-live entry, shown as a warning. A model can be one provider's current default while superseded elsewhere — Azure still defaults to gpt-4o — and the catalog records that rather than papering over it; kapi models list and the /models page both surface it.

Recommended vs known. The catalog is descriptive, not an allowlist: naming a model it does not list is never rejected — the string goes to the provider API as-is (the provider accepts or 404s it), and LimitsForModel simply falls back to a conservative batch size until an entry exists. So the catalog serves two audiences at once, and a recommended flag (absent = yes) separates them. It stays true for the models most projects should reach for, and is set false for a model that is fully supported but a poor default — capable-but-premium (Opus, Gemini Pro, o3), overkill for faithful content work, or off-task (Fable, tuned for creative writing). A non-recommended model keeps its ceilings and is callable by name; it just sits under "Advanced" on the /models page rather than in the primary list, which is sorted Recommended → Advanced → Legacy (superseded). An active default can never be non-recommended — a test enforces it.

Staying honest. Curation rots, so make check-models (scripts/modelcheck) is the alarm: it lists what each provider serves today and reports any catalogued model that is gone (remove it) or, with -candidates, any live model the catalog omits. The live half needs provider credentials and stays a manual or scheduled tool — a rate-limited provider must never be mistaken for a retired one, the same false-cliff trap the batch eval guards against. The keyless half — every published price must be for a catalogued model — is an ordinary unit test, so it runs in make test. The /models page is generated from the catalog through @neokapi/reference-data and gated by make check-reference-docs, so it cannot describe a model the catalog no longer lists. Refreshing the catalog itself is driven by scripts/prompts/update-model-catalog.md.

Each provider takes a Config struct with API key, base URL, model name, and generation parameters (temperature, max tokens, etc.). Azure OpenAI additionally accepts a TokenProvider function, enabling passwordless access via Azure Managed Identity.

StreamingLLMProvider​

An optional extension interface surfaces live progress events for providers that support them:

type StreamingLLMProvider interface {
    LLMProvider
    ChatStream(ctx context.Context, messages []Message,
        onEvent func(ChatStreamEvent)) (*ChatResponse, error)
    ChatStructuredStream(ctx context.Context, messages []Message,
        schema JSONSchema, onEvent func(ChatStreamEvent)) (*ChatResponse, error)
}

type ChatStreamEvent struct {
    Type    StreamEventType // StreamEventThinking | StreamEventContent | StreamEventDone
    Content string          // text chunk (thinking summary or output content)
    Usage   TokenUsage      // cumulative usage; populated on StreamEventDone
    Model   string          // model name; populated on StreamEventDone
}

The streaming methods deliver progress events through an onEvent callback and return the final aggregated *ChatResponse, rather than exposing a channel directly.

UIs and CLI tools display live thinking progress from providers that support it (Anthropic extended thinking, Gemini includeThoughts). A provider that does not implement StreamingLLMProvider can still be used — callers that need streaming check for the extension with a type assertion.

Concurrency model​

AI tools call the provider directly — provider.Translate() for a single block, provider.ChatStructured() for a batch. There is no intervening worker pool, rate limiter, or circuit breaker in the framework. Throughput is a property of the tool's own configuration, illustrated by translate (core/ai/tools/translate.go):

const (
    DefaultBatchSize        = 100
    DefaultBatchConcurrency = 1
)

AITranslateConfig exposes BatchSize and BatchConcurrency as schema fields, so they surface as CLI flags and flow config like any other tool option. The tool's Process method chooses a path from those values:

  • Block-by-block (batchSize <= 1 and concurrency <= 1) — the default BaseTool.Process drives one provider.Translate() call per translatable Block. Under a session it uses the simplest sequential skip/hydrate path (sessionHandleBlock): GetOverlay to skip already-translated Blocks, PutOverlay to write the result back. The batched path also honours session overlay caching, via processBatchedWithSession, which pre-filters cached Blocks and writes overlays on the way out.
  • Batched (processBatched) — drains all input Parts into a slice, selects the translatable Blocks (skipping already-translated ones when SkipMatched is set), groups them into batches of batchSize, and translates each batch in a single ChatStructured() call. Batches run under a chan struct{} semaphore sized to BatchConcurrency, so at most that many LLM calls are in flight at once. All Parts are then written downstream in their original order; entries missing from the structured response fall back to individual per-block translate() calls (one provider.Translate() per missing Block).

Streaming mode is orthogonal: when the provider implements StreamingLLMProvider and an OnProgress callback is supplied, the tool routes calls through ChatStream / ChatStructuredStream to surface live thinking summaries (see below). Transient-failure handling (retry, backoff) is left to the individual provider implementations and the underlying SDK; the framework does not impose a uniform retry policy.

This in-tool batching is distinct from the ParallelBlockTool concurrency in AD-004: Processing Engine, which parallelizes Part dispatch across the pipeline rather than grouping Blocks into a single API call.

AI tools​

AI capabilities reach the pipeline as ordinary Tools (AD-006: Tool System). On the CLI surface, translation is a single translate command across every backend, and QA a single qa command — the LLM is selected with --provider (the per-provider commands have collapsed into that one flag), while the underlying LLMProvider interface is unchanged:

ToolPurpose
translateTranslate untranslated Blocks using an LLM
qa --providerLLM-judged check of translations for fluency, accuracy, terminology
term-extractExtract terminology candidates from source Blocks
reviewReview translations with explanations
entity-extractExtract entities and term candidates (hybrid LLM + NER)

Because AI tools are ordinary Tools, they compose naturally:

sourcebindingchanrecyclememorychanterm-lookupchantranslateLLMProviderchanterm-enforcechanqaLLMProviderchansinkbinding

Terminology-aware prompts​

AI tools receive terminology context from upstream stages:

  • Term annotations — when term-lookup has run, matched terms and their preferred translations appear in the prompt.
  • Entity annotations — when entity-extract has run, identified entities (with DNT flags and locale formatting hints) appear in the prompt context.
  • Term constraints — a dedicated terminology section lists preferred and forbidden terms applicable to the current Block's domain, product, and market.

Terminology enforcement is not just a post-translation validation step; it actively guides AI translation quality from the start.

Structured batch output​

The batched translate path relies on ChatStructured() to make a multi-block response unambiguous. The tool sends a numbered prompt ([1] …, [2] …) and constrains the response to a JSON Schema that returns { translations: [{ index, text }] } with additionalProperties: false and strict: true. Index-text pairs eliminate the text-parsing ambiguity of free-form output and let the tool re-associate each translation with its source Block. Blocks whose source carries inline codes are rendered as placeholder-tagged text before the call and reconstructed from the response via ParseRunsPlaceholderText, so inline markup survives the round trip.

Prompt templates​

Prompt templates live in core/ai/prompt/ as versioned Go files using text/template:

  • translate.go — translation prompts with terminology and context (single and batched)
  • qa.go — quality assurance check prompts

Tool-specific prompts that have not been factored into the shared prompt package (e.g. the review prompt) are built inline in their tool, such as core/ai/tools/review.go.

Templates are context-aware: they include surrounding Blocks for document context, term constraints from term lookup, memory matches from leveraging, and format metadata (HTML tag handling instructions, CDATA boundaries, etc.).

Credential resolution​

AI providers read credentials at runtime from one of three sources:

  1. The CLI credential store (AD-013: Kapi CLI) — provider configs as JSON, API keys in the OS keychain.
  2. Environment variables — ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.
  3. Explicit --api-key flag on CLI invocation.

Flag overrides store overrides environment. API keys never appear in project files.

Default provider resolution​

Which provider a run uses when nothing names one is a separate question from how it authenticates, and it has exactly one resolver: config.ResolveAIDefault. It returns the provider, the model, and where the value came from (env | config | none), and config.SetAIDefault / ClearAIDefault are the matching single write path.

Precedence, matching every other config key: an explicit --provider/--model flag or an inline/recipe value first, then KAPI_AI_PROVIDER / KAPI_AI_MODEL, then the stored ai.provider / ai.model.

One resolver rather than six read sites, because six read sites had no shared answer to "what is configured, and where did it come from" — so a scope bug in the reader was invisible in all of them at once. Reporting the source alongside the value is what lets a diagnostic name the file or the environment variable to change, instead of asserting that nothing is configured.

The app config file is pinned to config.GlobalConfigFilePath() — the same function the writers use — never resolved through a search path. A recipe is project configuration and app config is per-machine, so the working directory is not a config location: a search path reaching it would load the recipe as the app config inside any project, since a kapi project's recipe is also named kapi.yaml, and every stored default would read as empty.

$HOME/.config/kapi/kapi.yaml and /etc/kapi/kapi.yaml are read as lower-precedence layers beneath the pinned file, so a hand-written config works. (On Linux the first of those is the pinned path, since os.UserConfigDir honours XDG; on macOS it resolves to ~/Library/Application Support, which is why the two differ at all.)

Scope boundary​

The framework's responsibility ends at the provider interface and the pipeline tools that call it. Server-side asynchronous job queues, multi-tenant quota enforcement, rate-limit budgets, and workspace-scale translation orchestration are a platform layer's concern, built on top of these framework primitives.

Consequences​

  • AI translation is a pipeline tool, not a separate system. It composes with all other tools without special orchestration.
  • Ordering is meaningful: memory leverage before AI translation avoids re-translating exact matches, reducing cost.
  • Terminology context flows through the pipeline via annotations, enabling AI tools to produce terminology-consistent translations from the start.
  • Throughput tuning lives on the tool, not in a hidden subsystem: a caller raises BatchSize to cut API call count and BatchConcurrency to run batches in parallel, with no separate worker pool to configure.
  • Structured batch output gives the tool a reliable index-text contract, so large documents translate in far fewer calls without parsing ambiguity.
  • Provider abstraction enables cost optimization: local Ollama for development, Claude or OpenAI for production.
  • Prompt templates are centralized and testable. The mock provider enables deterministic tests without API calls.
  • Azure Managed Identity eliminates API key management for production Azure deployments while the same interface continues to support key-based auth elsewhere.
  • ChatStructured gives tools a reliable JSON contract across providers with very different structured-output mechanisms.