stem_words

The stem_words template replaces its input with the Snowball stem of that word for the language of the configured LOCALE, and does nothing else — it does not split text into words. Stemming lets different inflections of a word match one another: with LOCALE = 'en', running and runs both index as run, so a query for one form retrieves documents written with the other. Snowball strips suffixes algorithmically, so runners stems to runner rather than to run, and ran — an irregular past form with no suffix to strip — stays ran.

One token goes in and exactly one token comes out. The stem replaces the original, which is not emitted alongside it, and no token is ever dropped: when the stemmer leaves a word as it is, or has no algorithm for the configured language, the token is the input unchanged.

On its own stem_words treats its whole input as a single token, so it is meant to receive pre-tokenized input. Place it as a stage inside a pipeline, after a word splitter such as split_text or split_text_csv, and after the normalize_tokens and remove_stopwords stages, so that it sees folded, filtered words.

As a function: stem_words(value, locale := '') — the value first, then the options in the order below. See tokenizer functions for how a value, a list and a chain of calls behave.

SQL

SELECT stem_words(['running', 'flies'], 'en_US.UTF-8') AS tokens;

 tokens
-----------
 {run,fli}

Options

Option	Type	Default	Description
LOCALE	string	required	Locale whose language selects the Snowball stemmer

LOCALE is the only option this template takes, and it has no usable default. Omitting it, or passing an empty string, leaves the locale unset and CREATE fails with stem_words: invalid locale; a value that is not a valid locale fails earlier with Invalid locale "<value>" for option "locale" (see locales), and a language without a Snowball stemmer fails with stem_words: there is no stemmer for the language "<language>" of locale "<name>". Only the language subtag reaches the stemmer, so 'en', 'en_US' and 'en_US.UTF-8' all select English, and the rest of the locale — region, encoding, keywords — is ignored. Any other tokenizer option — CASE, ACCENT, STOPWORDS — fails with stem_words(): unknown option "<name>". The feature flags are dictionary-level options, and all four of FREQUENCY, POSITION, NORM and OFFSET are supported here.

A stemmer is available for these languages: Arabic, Armenian, Basque, Catalan, Czech, Danish, Dutch, English, Esperanto, Estonian, Finnish, French, German, Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Lithuanian, Nepali, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Sesotho, Spanish, Swedish, Tamil, Turkish and Yiddish. A locale whose language is not in that list — 'zh', or 'C' — is still accepted at CREATE time; the dictionary then passes every token through unchanged, with no error and nothing filtered.

Tokenization

The template stems whatever token it receives. Applied to a single word it returns that word's stem. Several words separated by spaces are not split: the whole value is handed to the stemmer as if it were one word, so one token comes out rather than one per word — which is why stem_words is normally fed pre-tokenized input from a pipeline.

Input	LOCALE	Output tokens
running	en	run
running	zh (no stemmer)	running

On a standalone dictionary the token sits at the first position, and its offsets cover the whole input value rather than the shorter stem: the input running yields the token run with start offset 0 and end offset 7. An empty value still yields one token, which is empty.

stem_words does not fold case, strip accents or normalize Unicode, and it does not keep the unstemmed form next to the stem. For case-insensitive stems, lowercase upstream with a normalize_tokens step configured CASE = 'lower'; split_text takes the same option, and neither template folds case by default. The input is expected to be UTF-8 and nothing validates it.

SQL

CREATE TEXT SEARCH DICTIONARY stem_dict AS    stem_words('en');
SELECT ts_lexize('stem_dict', 'running');

 ts_lexize
-----------
 {run}

Stemming each word of a phrase

To stem every word in a phrase, split first and stem second. A pipeline that runs split_text_csv then stem_words reduces each token independently:

Input	Pipeline	Output tokens
running runners ran	split_text_csv (space) → stem_words (en)	run, runner, ran

SQL

CREATE TEXT SEARCH DICTIONARY stem_dict_words AS    split_text_csv(' ') | stem_words('en');
SELECT ts_lexize('stem_dict_words', 'running runners ran');

 ts_lexize
------------------
 {run,runner,ran}

See also

• split_text — the word splitter that usually precedes stem_words
• pipeline — chain a tokenizer before stem_words
• normalize_tokens — fold case before stemming, for case-insensitive stems
• CREATE TEXT SEARCH DICTIONARY