Relevance Scoring

Scorer Functions

Scorer functions compute a relevance score for each row matched by a full-text search predicate, measuring how well the row matches the @@ query in the same index scan. Each takes the index tableoid as its first argument and returns a FLOAT.

The score is an ordinary value — use it wherever you need one: project it in the SELECT list, rank with it in ORDER BY <scorer> DESC, filter on it in a WHERE comparison or fold it into a larger expression (for example, blending BM25 with business signals). Ranking with ORDER BY is the most common use, but it is not required.

Function	Description
BM25(tableoid[, k1, b])	Okapi BM25 relevance score — the recommended default.
TFIDF(tableoid[, with_norms])	Classic TF-IDF score.
lm_jm(tableoid[, lambda])	Language model, Jelinek-Mercer smoothing.
lm_dirichlet(tableoid[, mu])	Language model, Dirichlet smoothing.
indri_dirichlet(tableoid[, mu])	Indri-style Dirichlet smoothing.
dfi(tableoid[, measure])	Divergence-from-independence score (parameter-free).
raw_tf(tableoid)	Raw term frequency.
raw_boost(tableoid)	Raw query boost factor.
raw_dl(tableoid)	Raw document length.

Quick start

Filter with @@ to select matching rows, then rank them with a scorer in ORDER BY ... DESC. Add the primary key as a final sort key so ties resolve deterministically:

sql

SELECT id, BM25(docs_idx.tableoid) AS score
FROM docs_idx
WHERE body @@ ts_phrase('fox')
ORDER BY score DESC, id;

Every scorer follows this shape — swap BM25 for any function in the table above.

Requirements

Requirement	Applies to	What happens without it
frequency feature flag on the column	all scorers	Scoring is unavailable on the column.
norm feature flag on the column	lm_jm, lm_dirichlet, indri_dirichlet, dfi	The scorer returns 0 for every row.

Set both flags on the text search dictionary used by the indexed column:

sql

-- term frequency, needed by every scorer
-- document length norms, needed by lm_* and dfi
CREATE TEXT SEARCH DICTIONARY scored_en AS
    split_text() | stem_words('en_US.UTF-8')
    WITH (frequency, position, norm);

See token positions and feature flags for the full list. BM25, TFIDF, raw_tf, raw_boost and raw_dl need only frequency; the language-model scorers and dfi silently score 0 until norm is enabled.

One scorer per index per query

A single index scan can apply only one scorer function. Two different scorers over the same index in one SELECT raise Only one scorer function is allowed per inverted index. To compute several scores for the same rows, combine the per-scorer queries with UNION.

Choosing a scorer

If you want…	Use
A robust general-purpose default	BM25 — start here; tune k1 and b only if needed
A simple classic baseline / minimal tuning	TFIDF
Language-model ranking, short keyword queries	lm_dirichlet (length-aware smoothing)
Language-model ranking, longer / verbose queries	lm_jm (linear smoothing)
Indri / Lemur-compatible scores	indri_dirichlet
Good ranking with no parameters to tune	dfi
Raw signals to build your own score	raw_tf, raw_boost, raw_dl

In practice BM25 is the right default for almost all full-text ranking — it models term-frequency saturation (extra occurrences of a term add ever less) and document-length normalization, which plain TF-IDF does not. Reach for the language-model scorers (lm_*) when you want probabilistic query-likelihood ranking, dfi when you can't tune parameters and the raw_* features when you compose a custom relevance expression (for example mixing BM25 with business signals).

How they differ, simple to advanced:

• TF-IDF rewards a term that is frequent in the document and rare across the collection. It is linear in term frequency, so a few very common terms can dominate.
• BM25 is TF-IDF with two refinements: term-frequency saturation (k1) so the 10th occurrence counts far less than the 1st, and length normalization (b) so long documents are not unfairly favored for containing a term more times.
• Language models (lm_*) flip the question: instead of weighting terms, they estimate the probability that the document's word distribution would generate the query, smoothing each document with the whole-collection distribution so unseen terms do not zero out the score. They tend to track BM25 closely while exposing a single, interpretable smoothing knob.
• DFI is parameter-free: it scores a term by how far its observed frequency diverges from what statistical independence would predict, so there is nothing to tune at all.

Scorers

BM25(tableoid[, k1, b])

Signature. BM25(tableoid) -> FLOAT, BM25(tableoid, k1, b) -> FLOAT. Captures. Term-frequency saturation plus document-length normalization — the best all-round relevance signal.

The Okapi BM25 relevance score.

Parameter	Type	Default	Meaning
k1	FLOAT	1.2	Term-frequency saturation. Higher = extra occurrences keep mattering; lower = they saturate sooner.
b	FLOAT	0.75	Document-length normalization, in [0, 1]. b = 0 disables it (a.k.a. BM15); b = 1 fully normalizes by length.

How it works. A term contributes more when it appears often in a document and rarely across the collection, but each extra occurrence adds ever less (controlled by k1), and the contribution is scaled down for long documents (controlled by b). This is the standard relevance model behind Lucene, Elasticsearch and OpenSearch — start here and tune only if results need it.

SQL

SELECT id, BM25(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+------------
  2 | 0.38217354
  1 | 0.28062654
  3 | 0.19988364

Passing k1 and b explicitly tunes the score. With b = 0 (no length normalization) the two shorter documents are no longer penalized relative to each other and tie, where the default ranked the shorter one higher:

SQL

SELECT id, BM25(scored_idx.tableoid, 1.2, 0.0) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+------------
  2 | 0.38499752
  1 |  0.2449984
  3 |  0.2449984

TFIDF(tableoid[, with_norms])

Signature. TFIDF(tableoid) -> FLOAT, TFIDF(tableoid, with_norms) -> FLOAT. Captures. Term frequency weighted by inverse document frequency — a simple, classic baseline.

The classic tf–idf (term-frequency × inverse-document-frequency) score.

Parameter	Type	Default	Meaning
with_norms	BOOLEAN	false	Apply document-length normalization. Requires the norm flag on the column to have an effect.

How it works. Each matched term contributes tf × idf: more occurrences in the row and rarer occurrence across the collection both raise the score. It is cheaper than BM25 but has no term-frequency saturation, so a handful of very frequent terms can dominate — prefer BM25 unless you specifically want this baseline.

SQL

SELECT id, TFIDF(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+------------
  2 |  1.5870621
  1 | 0.91629076
  3 | 0.91629076

With with_norms = true the score is divided down for longer documents, which reorders nothing here but compresses the values:

SQL

SELECT id, TFIDF(scored_idx.tableoid, true) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+------------
  2 | 0.64791536
  1 | 0.45814538
  3 | 0.30543026

lm_jm(tableoid[, lambda])

Signature. lm_jm(tableoid) -> FLOAT, lm_jm(tableoid, lambda) -> FLOAT. Captures. Query-likelihood under a language model with fixed linear smoothing — suited to longer queries.

Query-likelihood language-model score with Jelinek-Mercer (linear) smoothing.

Parameter	Type	Default	Meaning
lambda	FLOAT	0.1	Smoothing weight in (0, 1]: each document's term probabilities are mixed with the collection's by lambda. Smaller favors precision on short queries; larger suits longer, verbose queries.

How it works. The model estimates the probability that the document would generate the query, mixing the document's own term distribution with the collection-wide distribution by a fixed fraction lambda. Requires the norm flag (returns 0 without it).

SQL

SELECT id, lm_jm(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+-----------
  2 | 3.1570003
  1 | 2.5055258
  3 | 1.7917594

lm_dirichlet(tableoid[, mu])

Signature. lm_dirichlet(tableoid) -> FLOAT, lm_dirichlet(tableoid, mu) -> FLOAT. Captures. Query-likelihood with length-adaptive smoothing — usually the best language model for short keyword queries.

Query-likelihood language-model score with Dirichlet smoothing.

Parameter	Type	Default	Meaning
mu	FLOAT	2000	Dirichlet prior. Larger = more smoothing (the collection prior dominates); set it near your average document length.

How it works. Like lm_jm, but the smoothing strength adapts to document length via the prior mu — short documents are smoothed proportionally more, which generally ranks short keyword queries better. Requires the norm flag (returns 0 without it).

SQL

SELECT id, lm_dirichlet(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+--------------
  2 | 0.0044765053
  1 | 0.0004988774
  3 |            0

indri_dirichlet(tableoid[, mu])

Signature. indri_dirichlet(tableoid) -> FLOAT, indri_dirichlet(tableoid, mu) -> FLOAT. Captures. Dirichlet-smoothed query-likelihood in the log domain, matching the Indri / Lemur search engine.

The Indri/Lemur variant of Dirichlet smoothing, without the score-floor clamp.

Parameter	Type	Default	Meaning
mu	FLOAT	2000	Dirichlet prior, as in lm_dirichlet.

How it works. Same smoothing as lm_dirichlet but scores are returned in the log domain (typically negative) and the low-score floor clamp is omitted, so values match those produced by the Indri search engine. Use it when you need Indri-comparable scores. Requires the norm flag (returns 0 without it).

SQL

SELECT id, indri_dirichlet(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+------------
  2 | -1.6049614
  1 |  -1.608939
  3 | -1.6114309

dfi(tableoid[, measure])

Signature. dfi(tableoid) -> FLOAT, dfi(tableoid, measure) -> FLOAT. Captures. How far a term's frequency in a document diverges from statistical independence — with nothing to tune.

Divergence-from-independence term weighting.

Parameter	Type	Default	Meaning
measure	VARCHAR	'standardized'	Divergence statistic. One of 'standardized', 'saturated', 'chi_squared'.

How it works. For each term the model computes the frequency expected under independence, then scores by how far the observed frequency diverges from it — the measure selects which divergence statistic to use. It is parameter-free (no k1, b, lambda or mu), making it a strong choice when you can't or don't want to tune. Requires the norm flag (returns 0 without it).

SQL

SELECT id, dfi(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+------------
  2 |  1.4022679
  1 | 0.29114002
  3 |          0

The 'saturated' and 'chi_squared' measures rank the same documents here but produce different score magnitudes:

SQL

SELECT id, dfi(scored_idx.tableoid, 'saturated') AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+-------------------
  2 | 1.321928
  1 | 0.3219280(varies)
  3 | 0

SQL

SELECT id, dfi(scored_idx.tableoid, 'chi_squared') AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+-------------
  2 |   1.8875252
  1 | 0.070389315
  3 |           0

raw_tf(tableoid)

Signature. raw_tf(tableoid) -> FLOAT. No parameters. Captures. The raw count of matched-term occurrences in each row — a building block, not a ranking model.

Raw term frequency of the matched terms in each row. Use it inside a custom relevance expression; it does not normalize for length or rarity.

SQL

SELECT id, raw_tf(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+-------
  2 |     3
  1 |     1
  3 |     1

raw_boost(tableoid)

Signature. raw_boost(tableoid) -> FLOAT. No parameters. Captures. The query-time boost factor that applied to each match (see Boosting below).

Raw query boost contribution for each row. With no ^ boost in the query every match returns 1 — equivalent to a constant score (see cross-engine notes). When a clause is boosted with ^ f, the matched rows carry that factor f.

SQL

SELECT id, raw_boost(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+-------
  1 |     1
  2 |     1
  3 |     1

raw_dl(tableoid)

Signature. raw_dl(tableoid) -> FLOAT. No parameters. Captures. The length of the matched column for each row — the normalization input that BM25(b>0) and the lm_*/dfi scorers use internally.

Raw document length of the matched column: the number of token positions when the column records POSITION, so tokens stacked at one position, such as synonyms, count once, and the number of tokens otherwise. See feature flags.

SQL

SELECT id, raw_dl(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ ts_phrase('fox')ORDER BY score DESC, id;

 id | score
----+-------
  3 |     9
  2 |     6
  1 |     4

Boosting

The ^ operator multiplies a query clause's contribution to the score, so you can weight some clauses above others. The factor flows straight through every scorer: boosting ts_phrase('fox') ^ 3.0 multiplies each matched row's BM25 score by exactly 3.

SQL

SELECT id, BM25(scored_idx.tableoid) AS scoreFROM scored_idxWHERE body @@ (ts_phrase('fox') ^ 3.0)ORDER BY score DESC, id;

 id | score
----+------------
  2 |  1.1465207
  1 | 0.84187967
  3 |   0.599651

raw_boost exposes the applied factor directly. See Relevance ranking → Boosting for boosting across multiple columns.

Per-node score control

Two modifiers change how a part of a query scores, while the rest of it stays on the ORDER BY scorer. Both are TSQUERY modifiers, like ::boost and ::slop, so they travel with the value: they apply to a parenthesized predicate, to a group built by ts_any or ts_all, and to a bound parameter.

::merge(policy)

Sets how a boolean node combines what its branches scored.

Policy	Effect
'sum' · 'default'	Add every matching branch's contribution. This is the behaviour without the modifier.
'max'	Take the best matching branch and ignore the others.

The name is read case-insensitively. Any other name is an error rather than a silent fallback.

sql

SELECT id, BM25(docs_idx.tableoid) AS score
FROM   docs_idx
WHERE  (body @@ 'fox' OR body @@ 'cat')::merge('max');

Under the default sum, a document containing both fox and cat collects both contributions and outranks a document that matched one branch strongly. Under max it is scored once, on its best branch.

The synonym case. This is what makes max worth reaching for. Expanding a user's car into car OR automobile creates two branches that mean one thing, so summing them rewards a document for happening to use both spellings:

sql

SELECT id, BM25(docs_idx.tableoid) AS score
FROM   docs_idx
WHERE  body @@ ts_any(['car', 'automobile'])::merge('max');

Synonyms that the column's dictionary produces itself, with expand_solr_synonyms or expand_wordnet_synonyms, already score this way: the tokens a query word expands to at one position form one group merged with max, in bare strings, ts_tokenize, plainto_tsquery, websearch_to_tsquery, to_tsquery, ts_any/ts_all over ts_tokenize, and a one-word ts_phrase.

It binds to its own node. Nesting is therefore meaningful. With ((a OR b)::merge('max') OR c) the inner group takes the best of a and b and the outer group still adds c on top. Move the modifier to the outer group and the best single branch of all three wins.

It requires a boolean group. A leaf has no branches to combine, so (body @@ 'fox')::merge('max') is an error.

EXPLAIN prints Merge: max on the node that carries the policy. Same-field branches fuse into one Terms node, which carries the policy of the Or it replaced.

::score(scorer)

Scores the wrapped subtree with a scorer of its own. Everything outside it stays on the ORDER BY scorer, so one scan can mix signals:

sql

SELECT id, BM25(docs_idx.tableoid) AS score
FROM   docs_idx
WHERE  body @@ 'fox' OR (body @@ 'cat')::score('constant(1)');

The fox branch stays on BM25 and the cat branch scores a flat 1. Any scorer may be named.

::score(NULL) excludes a subtree from scoring altogether. It still selects rows, contributes nothing to the score and requests no index features, so its postings are read without frequency or norms:

sql

SELECT id, BM25(docs_idx.tableoid) AS score
FROM   docs_idx
WHERE  body @@ 'fox' OR (body @@ 'cat')::score(NULL);

Rows that match only through the unscored branch come back with 0, and the rest score exactly as if that branch were not in the query. This is the equivalent of an Elasticsearch filter clause.

Zero is a real score

An unscored branch makes 0 a legitimate score, which matters for ORDER BY <scorer> DESC LIMIT k. The top-k collector admits zero-scoring hits, so those rows are returned rather than dropped.

Combining the two

The modifiers compose in either order, and ::score wraps the group while ::merge applies to the node whose branches actually combine:

sql

-- a constant score that stays constant: a document matching both
-- branches scores 42, not 84
WHERE (body @@ 'fox' OR body @@ 'cat')::score('constant(42)')::merge('max')

A per-branch scorer competes on its own terms under a group policy, so a constant(10) branch survives a max against a BM25 sibling. An unscored branch has nothing to compare, so the other branch wins by default.

Top-K and WAND pruning

The common shape ORDER BY <scorer>(idx.tableoid) DESC LIMIT k returns the best k matches. Building the index with the optimize_top_k option enables WAND pruning, which skips candidates that provably cannot reach the top k:

sql

CREATE INDEX docs_idx ON docs
    USING inverted (id, body scored_en)
    WITH (optimize_top_k = 'bm25(1.2, 0.75)');

Pruning engages only when the ORDER BY scorer can use the score bounds the index stores for its optimize_top_k scorer and the filter is a single term, an OR of terms that any one term satisfies, an AND of terms or of such ORs (the latter when enough rows match), terms with AND NOT exclusions, one required term with optional scoring terms (+fox quick), or a phrase without slop. An OR prunes only when the scores of its terms add up: one that keeps the best score, such as a fuzzy match that expands to several terms or ::merge('max'), does not. Otherwise the query still runs correctly, just without the optimization. EXPLAIN shows Top: k, optimized on the scan when the scorer qualifies, whatever the filter. See Relevance ranking → Top-K queries and WAND pruning for the full conditions, including which scorers share score bounds.

Cross-engine notes

If you are coming from Elasticsearch or OpenSearch, here is how their relevance-tuning concepts map onto SereneDB (the left column links to the Elasticsearch reference):

Elasticsearch / OpenSearch	SereneDB
_score (implicit relevance)	any scorer over tableoid, e.g. BM25(idx.tableoid)
boosting query / per-clause "^2"	the ^ operator: ts_phrase('fox') ^ 2.0 (Boosting)
constant_score	raw_boost(idx.tableoid) returns 1 for every match when no ^ boost is applied; or ORDER BY a literal
function_score weight	fold the scorer into an arithmetic expression, e.g. BM25(idx.tableoid) * 2
function_score field_value_factor	blend the scorer with a column in the SELECT / ORDER BY expression
dis_max	::merge('max') on the disjunction (Per-node score control)
bool filter clause	::score(NULL) on the clause (Per-node score control)
Top-K acceleration	optimize_top_k + WAND pruning (Top-K)
Tie-breaking	extra ORDER BY columns, typically the primary key
Reciprocal Rank Fusion	Reciprocal Rank Fusion

SereneDB has no single function_score-style query type. Because a scorer is just a FLOAT-valued expression, you compose the same effects directly in SQL — multiply, add, threshold in WHERE, or blend with table columns — rather than through a dedicated DSL.

See also

• Full-Text Search Functions — operators, constructors, parsers
• Inverted Index · Relevance ranking guide
• Text Analysis — the frequency and norm flags scoring requires
• BM25/TFIDF Ranking recipe — parameter tuning