Tokenizer Functions
Every template of CREATE TEXT SEARCH DICTIONARY is also a scalar function of the same name: split_text, stem_words, generate_ngrams. The function takes the value to analyze first and then the template's own arguments — the same names, the same positional order, the same defaults as on the template's page — and returns the tokens as a list. It is an implicit instantiation of the tokenizer: split_text(body, case := 'lower') in a query produces exactly what a dictionary created AS split_text(case := 'lower') produces for body, so the functions let you analyze text ad hoc, preview a dictionary stage by stage, or tokenize a column without creating anything in the catalog. WITH (help) on CREATE TEXT SEARCH DICTIONARY prints the function form under every template.
SELECT split_text('The Quick fox-trot.', case := 'lower') AS tokens; tokens---------------------- {the,quick,fox,trot}Values, lists and chains
| Input | Behaviour |
|---|---|
VARCHAR | The value is analyzed as one value, the way a dictionary analyzes one cell. |
VARCHAR[] | Each element is analyzed on its own and the results are concatenated into one list, so a function applied to another function's result acts as the next stage of a pipeline. |
VARCHAR[] for generate_shingles and generate_wildcard_ngrams | The list is one token stream: the elements are the base tokens the shingles or wildcard grams are built from. |
NULL | A NULL value yields NULL; NULL elements of a list are skipped. |
VARCHAR[N] | A fixed-size array is read as the list it is, in the value position and in a list-valued option. |
Nesting the functions is the pipeline: the inner call produces the list the outer call re-analyzes element by element, so stem_words(normalize_tokens(split_text(v, case := 'lower'), 'en_US.UTF-8', accent := false), 'en_US.UTF-8') is the dictionary split_text(case := 'lower') | normalize_tokens('en_US.UTF-8', accent := false) | stem_words('en_US.UTF-8'):
SELECT stem_words(normalize_tokens(split_text('The Runners Café', case := 'lower'), 'en_US.UTF-8', accent := false), 'en_US.UTF-8') AS tokens; tokens------------------- {the,runner,cafe}CREATE TEXT SEARCH DICTIONARY english AS split_text(case := 'lower') | normalize_tokens('en_US.UTF-8', accent := false) | stem_words('en_US.UTF-8');
SELECT stem_words(normalize_tokens(split_text('The Runners Café', case := 'lower'), 'en_US.UTF-8', accent := false), 'en_US.UTF-8') = ts_lexize('english', 'The Runners Café') AS same; same------ tA list literal is analyzed element by element:
SELECT normalize_tokens(['Café', 'ÉCOLE'], 'en_US.UTF-8', case := 'lower', accent := false) AS tokens; tokens-------------- {cafe,ecole}generate_shingles and generate_wildcard_ngrams need a token stream, so they take the output of another function; a bare VARCHAR counts as a single token:
SELECT array_length(generate_shingles(split_text_csv('quick brown fox', ' '), 2, 2), 1) AS terms; terms------- 5A list carries no token positions, only order, so the two wrappers read every element as one position and treat neighbours as adjacent. A dictionary sees the real positions instead, which differ wherever a stage emits several tokens at one position: the grams of generate_ngrams are alternatives for the same word, and the expansions of expand_solr_synonyms are alternatives for the same token. Shingling such a stage therefore belongs in a dictionary: the function generate_shingles(generate_ngrams(v, 2, 3), 2, 2) pairs each gram with the next one in the list, where the dictionary generate_shingles(generate_ngrams(2, 3), 2, 2) pairs across positions and leaves same-position grams unpaired.
The value may come from a column; the options may not. Every option is folded to a constant when the query is bound, because the tokenizer is built once per query, so an option that varies per row fails with <name>(): option "<option>" must be a constant. A NULL option leaves it at its default, and a misspelled or missing option fails to bind with No function matches the given name and argument types, followed by the candidate signatures.
An option whose value is a list of strings takes either spelling of the dictionary form: the list, remove_stopwords(v, stopwords := ['the']), or the string its template page documents, remove_stopwords(v, stopwords := '"the"') and encode_geopoint(v, latitude := 'loc/lat'). The candidate signature spells that out as UNION(str VARCHAR, list VARCHAR[]), which a fixed-size VARCHAR[N] also satisfies; any other type fails to bind.
CREATE TABLE articles (id INTEGER, body VARCHAR);
INSERT INTO articles VALUES (1, 'The Runners'), (2, 'were Running');
SELECT id, stem_words(split_text(body, case := 'lower'), 'en_US.UTF-8') AS tokens FROM articles ORDER BY id; id | tokens----+-------------- 1 | {the,runner} 2 | {were,run}SELECT split_text(body, case := body) FROM articles;error must be a constantThe result type follows the tokenizer: VARCHAR[] for every text template, BLOB[] for collate_tokens, encode_geopoint and encode_geojson, whose terms are binary. A template that stores a per-document payload alongside its terms — generate_shingles with store_tokens, encode_geojson with a non-source coding — returns only the terms.
Four templates have no function form, because SQL already expresses them: keyword is the list literal [value], pipeline is nesting, union is list_concat of two calls, and a sql stage is the expression itself.
A stored dictionary has no function form: it is an analyzer instance rather than a template, so a query runs it through ts_lexize and an analyzer expression names it as a stage.
Functions
Each signature lists the value first and then the template's options with their defaults; a required option has none. Option semantics, constraints and error messages are on the page each row links to.