Text Analysis

Text analysis turns the free-form text of a column into the sequence of tokens that an inverted index actually stores and searches. Getting analysis right is what lets a search for running shoes find a document that says Run faster in our Shoes! — the surface forms differ, but they analyze to the same tokens.

Analysis in SereneDB is configured entirely through a text search dictionary attached to a column. A dictionary is assembled from templates: some templates tokenize (split text into tokens), others normalize (lowercase, fold accents, stem, drop stop words), and the pipeline template composes them. There is no separate "token filter" object — every stage is a template.

Tip

Token vs. term. A token is produced during analysis and carries metadata (its position, byte offsets). A term is the normalized value that actually lands in the index. Tokens are what the pipeline transforms; terms are what you match against.

The same analysis at index time and query time

The single most important rule: a column's dictionary is applied both when the column is indexed and when a query runs against it. The data and the query pass through the identical pipeline, so their tokens line up.

Without analysis, a literal comparison of the query FOX against the stored text Quick BROWN Fox would not match. After analysis both sides reduce to the token fox, and the lookup succeeds. You can preview exactly how a dictionary tokenizes any string with ts_lexize — use it whenever you are tuning a dictionary:

Query

CREATE TEXT SEARCH DICTIONARY ta_text AS    split_text(case := 'lower') | normalize_tokens('en_US.UTF-8', accent := false);
SELECT ts_lexize('ta_text', 'Quick BROWN Fox');

Result

 ts_lexize------------------- {quick,brown,fox}

Overriding the query-time analyzer

Symmetry is the default, not a hard rule. To analyze a query string with a different dictionary than the column's — Elasticsearch's search_analyzer — wrap it in ts_tokenize(text, 'dict') or the 'text'::tokenize('dict') cast. A common use is forcing exact matching against an otherwise-stemmed column with '…'::tokenize('keyword').

Tokenizing templates

The tokenizing template decides how text is split. The most common is split_text, which splits on word boundaries (shown above). Others target specific needs:

A verbatim column — the keyword template, or simply a column with no dictionary — emits the whole value as a single token, giving exact, case-sensitive matching for ids, codes and categories:

Query

CREATE TEXT SEARCH DICTIONARY ta_keyword AS keyword() WITH (frequency);
SELECT ts_lexize('ta_keyword', 'Hello World');

Result

 ts_lexize----------------- {"Hello World"}

The generate_ngrams template emits overlapping character n-grams, which power substring and fuzzy matching:

Query

CREATE TEXT SEARCH DICTIONARY ta_ngram AS    generate_ngrams(2, 3);
SELECT ts_lexize('ta_ngram', 'cat');

Result

 ts_lexize------------- {ca,cat,at}

Further tokenizing templates — generate_sparse_ngrams, split_text_csv / split_by_delimiters, split_text_icu, split_by_non_alpha, split_by_pattern, expand_path, generate_wildcard_ngrams, generate_shingles, sql — are listed in the CREATE TEXT SEARCH DICTIONARY reference.

Normalization

Normalization rewrites tokens so that equivalent forms collapse together. Each normalizer is a stage chained after the split_text split:

Stemming reduces words to a root form, improving recall by matching different inflections:

Query

CREATE TEXT SEARCH DICTIONARY ta_stem AS    split_text(case := 'lower') | normalize_tokens('en_US.UTF-8', accent := false) | stem_words('en_US.UTF-8');
SELECT ts_lexize('ta_stem', 'running runners ran');

Result

 ts_lexize------------------ {run,runner,ran}

Stop words drop high-frequency words that carry little meaning:

Query

CREATE TEXT SEARCH DICTIONARY ta_stopwords AS    split_text(case := 'lower') | normalize_tokens('en_US.UTF-8', accent := false) | remove_stopwords(['the', 'of']);
SELECT ts_lexize('ta_stopwords', 'the speed of light');

Result

 ts_lexize--------------- {speed,light}

Accent folding maps accented characters to their ASCII base so café matches cafe. It is the normalize_tokens stage's job — normalize_tokens('en_US.UTF-8', accent := false) folds accents away, and without that stage they are preserved:

Query

CREATE TEXT SEARCH DICTIONARY ta_accent AS    split_text(case := 'lower') | normalize_tokens('en_US.UTF-8', accent := false);
SELECT ts_lexize('ta_accent', 'Café');

Result

 ts_lexize----------- {cafe}

Case folding (split_text(case := 'lower')) is applied in every example above. collate_tokens is the other normalizing stage, producing sort keys for a locale.

Locale-aware analysis. The collate_tokens and normalize_tokens templates take an ICU locale, so sorting and equality follow a language's rules rather than raw byte order — German de, for example, sorts ä next to a. A collate_tokens dictionary turns each value into one locale-ordered key, which is ideal for range queries and exact ordering on a column. The same ICU locales back the SQL COLLATE clause.

Composing with pipeline

A chain of stages joined with | is a pipeline: each stage re-analyzes the tokens of the one before it. Here a split_text_csv tokenizer splits on commas, then a normalize_tokens stage lowercases each token:

Query

CREATE TEXT SEARCH DICTIONARY ta_pipeline AS    split_text_csv(',') | normalize_tokens('en_US.UTF-8', case := 'lower');
SELECT ts_lexize('ta_pipeline', 'RED,Green,BLUE');

Result

 ts_lexize------------------ {red,green,blue}

Token positions and feature flags

By default the index records only which terms appear in which rows. Some query and ranking features need extra per-token information, enabled with feature flags on the dictionary (or per-column in the index):

Flag	Records	Needed for
frequency	how often each term occurs	relevance scoring
position	each token's ordinal position	phrase and proximity queries
offset	each token's byte offsets in the source value	highlighting
norm	a length-normalization factor	some scorers

The flags have dependencies: position and norm require frequency, and offset requires frequency and position. Positions are what let phrase search distinguish quick brown fox from fox brown quick — the tokens are the same, but their positions differ:

Enable only the flags your queries need — each one enlarges the index.

Tip

Analyzed vs. verbatim (term-level). A column with a dictionary is analyzed — matching is token-based and tolerant of form. A verbatim column is not analyzed — matching is exact. This mirrors the full-text vs. term-level distinction in other search engines: use analyzed columns for natural language, verbatim columns for exact values.

See also

• CREATE TEXT SEARCH DICTIONARY — every template and option
• Inverted Index — operator classes and how dictionaries attach to columns
• Full-Text Search · Ranking
• Full-Text Search Functions — ts_lexize and the query functions