Skip to main content

Edit this page

classify_text

The classify_text template runs a pre-trained text-classification model over the input and emits its predicted category labels as tokens. This lets you index documents by an inferred property — topic, language, sentiment — and then query them by label like any other term, without storing the label yourself.

At index and query time the analyzer hands the value to a fastText supervised model loaded from MODEL_LOCATION and turns the model's predictions into terms. TOP_K caps how many of the highest-scoring labels are emitted and THRESHOLD drops any whose probability falls below the given score.

The model file is required and must exist on the server host when the dictionary is created; a file the loader rejects fails with failed to load fasttext model from: <path>, error: <...> and no dictionary is created. It also has to be a supervised model: a model trained any other way still loads, but classifying a value with it fails with fastText's Model needs to be supervised for prediction!. Dictionaries that name the same MODEL_LOCATION share one in-memory copy of the model.

Classification and find_nearest_words are the two model-backed templates — use classify_text to tag a document with predicted categories, and find_nearest_words to expand it with semantically related terms.

As a function: classify_text(value, model_location := '', top_k := 1, threshold := 0) — the value first, then the options in the order below. See tokenizer functions for how a value, a list and a chain of calls behave.

Query
SELECT classify_text('Which baking dish is best to bake a banana bread?', model_location := '${RESOURCES}/tests/iresearch/model_cooking.bin', top_k := 2) AS labels;
Result
 labels------------------------------------ {__label__bananas,__label__baking}

Options​

OptionTypeDefaultDescription
MODEL_LOCATIONstring — a path on the server hostrequiredPath to the fastText model file. Its existence is checked when the dictionary is created, so a path that is not there fails with File "<path>" referenced by option "model_location" does not exist; omitting the option altogether fails with classify_text: empty model location
TOP_Kinteger1Maximum number of highest-scoring labels emitted per value. Must be greater than zero, otherwise classify_text: top_k must be positive
THRESHOLDdouble0.0Minimum probability a label must reach to be emitted. Must lie in [0.0, 1.0], otherwise invalid value in "threshold" parameter. Should be in [0, 1]. The default keeps every label the model returns

Tokenization​

The value goes to the model as one line. The model's own reader splits it on ASCII whitespace only — space, tab, carriage return, vertical tab, form feed and NUL — and looks each word up in the model's dictionary; a word the model does not know contributes nothing. A newline is where that line ends: the reader stops at the first \n in the value, so anything after it is never classified. Nothing is folded on the way in: no case conversion, no accent folding, no Unicode normalization, so the input bytes reach the model unchanged.

The emitted tokens are the model's label strings, copied verbatim from the model file, which means the label prefix (__label__, unless the model was trained with another one) is part of the term. Labels arrive ordered by descending probability, capped at TOP_K, and a label is kept only when its probability is at least THRESHOLD, so one value yields between zero and TOP_K tokens. The input text itself is never emitted.

All labels of one value share a single position: they form one stacked set rather than a sequence, so a phrase or proximity query cannot order them. Each label also carries the offsets of the whole value — start 0, end the length of the value in bytes — because a predicted label has no span of its own in the text.

An empty value, or a value in which the model recognizes no word, produces no tokens at all. All four feature flags — FREQUENCY, POSITION, NORM and OFFSET — are accepted with this template, as long as their dependencies hold: OFFSET requires POSITION, and POSITION and NORM require FREQUENCY.

Examples​

Point MODEL_LOCATION at a trained fastText classifier and choose how many labels to keep. This dictionary emits at most three labels per value and drops any whose probability is below 0.5:

Query
CREATE TEXT SEARCH DICTIONARY classifier AS    classify_text(model_location := '/models/cooking.bin', top_k := 3, threshold := 0.5);

With the dictionary attached to a text column in a USING inverted index, each document is indexed under its predicted labels, so a query for a label matches every document the model assigned it. Which labels those are is the model's business alone: the model file fixes both the label set and the prefix carried by every term.

See also​