Skip to main content

Learn kapi › Explore the engine

Segmentation

Compare the sentence splitters on the cases that trip them up.

Before text can be translated well it has to be split into sentences correctly: an abbreviation, a price, a quotation, a script with no spaces. The engine offers rule-based, Unicode, hybrid and learned segmenters. This lab runs them side by side on the tricky cases and shows where they agree, as overlays on the same blocks.

  1. Abbreviations and decimalsDr. Smith, $3.50, Jan. 5: every full stop that is not a sentence end. The four instant engines run as soon as the chapter opens; the colours mark boundaries two or more engines drew in the same place.
  2. Dialogue and quotesA question mark inside quotation marks, then a comma: whether the quote closes the sentence is exactly where rule sets differ.
  3. No spaces at allJapanese marks sentence ends with its own punctuation and no spaces. The locale is set with the sample, so each engine applies its rules for the language rather than for English.
  4. Paragraphs are blocksTwo paragraphs become two blocks before any engine runs: the content model's structure comes first, and segmentation works inside each block.
  5. A learned model, on requestSaT and the local language models are opt-in: select one in the engine list and run again. Each downloads its weights the first time, and the download shows in the navbar as the plugin loads.