0.13.18 — a snippet to size

From the start, semantic search has shown under each title the paragraph that decided the match — so the reader can see why they got this text. But the index computes vectors from whole paragraphs, and paragraphs vary a lot: from a single sentence to more than a dozen. The longest exceed a thousand characters. Three such results mean several phone screens before the reader sees the second title.

Since 0.13.18 the snippet is at most 300 characters. The cut follows sentences, not characters: whole sentences go in up to the limit, because a sentence broken mid-word reads like a bug. Only a single sentence longer than the limit is cut at a word boundary. An ellipsis marks the side where something was left out.

That leaves the question of which sentences to show. Search works on meaning and does not know which sentence of the paragraph tipped the balance. But when the query and the paragraph share a word stem — as in the Polish “masoneria” and “masońskim” — that sentence is the best evidence of the match, so the window starts there. The stem is the first five letters after stripping diacritics, computed only for words of four letters or more. Without a shared stem the window starts at the beginning of the paragraph.

The index and the vectors do not change. Trimming happens at response time, so no rebuild is needed, and the limit is a single engine constant, `FRAGMENT_MAX`.