find_nearest_words
The find_nearest_words template replaces each word of its input that the model knows with the words whose vectors lie closest to it in a pre-trained fastText model. A document indexed through this template can be found by related words it never literally contained — a recall-oriented complement to exact full-text matching. The document's own words are not indexed as such: fastText leaves a word out of its own neighbor list, and this template emits nothing but neighbors, so an input word reaches the index only where the model lists it as a neighbor of another word in the same value.
As a function: find_nearest_words(value, model_location := '', top_k := 1) — the value first, then the options in the order below. See tokenizer functions for how a value, a list and a chain of calls behave.
SELECT find_nearest_words('salt', model_location := '${RESOURCES}/tests/iresearch/model_cooking.bin', top_k := 2) AS neighbours; neighbours--------------------- {homogenized,teach}How it works
The model is loaded from MODEL_LOCATION when the dictionary is created, and dictionaries naming the same path share one in-memory copy of it. For each word of a value that the model recognizes the analyzer asks for that word's TOP_K nearest words by cosine similarity and emits them as terms. Applied at index time this decides what a document can match; applied to the query it decides what the query reaches. For example the small cooking model used in the example below turns "salt" into homogenized and teach.
The model file is required and must be reachable from the server process at the path given in MODEL_LOCATION; the dictionary cannot be created without a loadable model, and a file the loader rejects fails with failed to load fasttext model from: <path>, error: <...>. Neighbors are looked up per whole word, so the model's dictionary has to hold plain words only: a model carrying character n-grams — minn 3 and maxn 6, what skipgram and cbow train with by default — or word n-grams (wordNgrams above 1) is not supported. Training with fastText's supervised mode produces such a model, because that mode sets minn and maxn to 0; the cooking model used in the example below is one, and it is the same kind of model classify_text needs. Where classify_text tags a document with predicted category labels, find_nearest_words indexes it under related words instead of its own.
Options
| Option | Type | Default | Description |
|---|---|---|---|
MODEL_LOCATION | string — a path on the server host | required | Path to the fastText model file |
TOP_K | integer | 1 | Number of nearest neighbors emitted per recognized word |
MODEL_LOCATION has no usable default. The path is checked on the server process when the statement runs: a path that is not there — an empty string included — fails with File "<path>" referenced by option "model_location" does not exist, and omitting the option altogether fails with find_nearest_words: empty model location. TOP_K must be greater than zero — 0 or a negative value fails with find_nearest_words: top_k must be positive. Any other option — THRESHOLD, for instance, which belongs to classify_text — fails with find_nearest_words(): unknown option "<name>". The index feature flags are dictionary-level options and all four of FREQUENCY, POSITION, NORM and OFFSET are supported here, as long as their dependencies hold: OFFSET requires POSITION, and POSITION and NORM require FREQUENCY.
Tokenization
The value is split into words by the model's own reader, which breaks on ASCII whitespace only — space, tab, carriage return, vertical tab, form feed and NUL. A newline ends the reader's line as well as the word before it: the first \n in a value becomes fastText's end-of-sentence marker </s>, and the words after it are never looked up. That marker is itself a dictionary entry in a model trained on multi-line text, so it is looked up like a word and contributes its own neighbors. Each word the model knows is looked up in its word vectors, and its TOP_K closest words are emitted in order of descending similarity. A word the model does not know contributes nothing, and neither does a word whose neighbor list comes back empty. An empty value yields no tokens at all.
All neighbors of one input word share a single position, and the position counter advances only for input words that produced at least one neighbor. Every token carries the offsets of the whole input value — start 0, end the value's length in bytes — rather than the offsets of the word it came from. No case folding, accent folding or Unicode normalization is applied; the input bytes reach the model unchanged.
Which neighbors come out is entirely the model's business, and they are raw model vocabulary, so they can carry punctuation. With the small cooking model used in the example below:
| Input | TOP_K | Tokens |
|---|---|---|
salt | 1 | homogenized at position 1 |
salt | 2 | homogenized, teach, both at position 1 |
salt oil | 2 | homogenized, teach at position 1; tube", "breather at position 2 |
Examples
Point MODEL_LOCATION at a trained fastText model and choose how many neighbors to emit per word:
CREATE TEXT SEARCH DICTIONARY nn_dict AS find_nearest_words(model_location := '/models/cooking.bin', top_k := 2);Attached to a text column in a USING inverted index, the dictionary indexes each document under the nearest neighbors of its words, widening recall.
See also
classify_text— tag text with predicted category labelsexpand_solr_synonyms— expand terms through a synonyms map you write yourself- Full-Text Search
- CREATE TEXT SEARCH DICTIONARY