Hybrid Search with SereneDB
Hybrid search here is one index and one SQL query: BM25 over the text column and ANN over the embedding column, in the same statement.
No separate vector database, no sync pipeline between two systems, and nothing to keep consistent — the two signals are columns in one inverted index.
What Is Hybrid Search?
Hybrid search runs two retrieval signals over the same corpus and combines their results. Keyword search — BM25 over a full-text search index — matches the words the user actually typed and ranks by term statistics. Vector search compares embeddings and ranks by distance, so it matches meaning rather than spelling. A hybrid search engine keeps both and merges the two ranked lists into one.
The example from our own cookbook: a shopper searches running. Keyword search returns trail running shoe and road running shoe and misses marathon racer entirely — the word is not there. Vector search ranks marathon racer second by distance, because the embedding knows what a marathon is. Hybrid search returns all three, with the two documents that scored on both signals at the top.
Why Hybrid Search Matters
Each signal on its own has a failure mode that the other one does not have. A hybrid search engine exists because those two failure modes do not overlap.
The keyword gap
Keyword matching does not know synonyms or meaning. A search for laptop does not find notebook computer; a search for running does not find marathon racer. The query was correct and the document exists, and the user still gets zero results — which they read as "the feature is not there".
The vector gap
Vector search always returns something, and near-neighbours are not always relevant. Ask for the SKU AB-1234 and an embedding has no idea what that string means — it returns ten plausible products, none of them the one. Exact matches on identifiers, error codes and function names are exactly what distance loses.
How hybrid search closes both
Either one signal constrains and the other ranks — keyword filters to the exact matches, vector orders them by meaning — or both signals rank and their ranks are fused. Under Reciprocal Rank Fusion a document that placed well on both lists ends up above a document that only placed well on one.
How SereneDB Does Hybrid Search
As a hybrid search database SereneDB has one structure to explain: a single inverted index over one table covers the text column for BM25, the vector column for IVF ANN, and any verbatim or structured columns you filter on. Both query strategies below read that same index.
One index, two signals
One CREATE INDEX … USING inverted() names the text column with a dictionary for BM25 and the vector column with ivf for ANN. It is one index over one table, not two services with a pipeline between them.
CREATE INDEX items_idx ON items
USING inverted (id, name en, emb ivf (metric = 'l2'));Filtered ANN
One signal is a hard filter, the other ranks. The shopper wants shoes and nothing else, ordered by semantic closeness: a full-text predicate constrains the candidate set, and vector distance sorts what survives.
SELECT id, name FROM catalog_idx
WHERE name @@ ts_phrase('shoes')
ORDER BY emb <-> [1.0, 0.0, 0.0]::FLOAT[3]
LIMIT 2;Reciprocal Rank Fusion (RRF)
Both signals rank. BM25 produces one ordered list, vector distance another, and the ranks are fused as 1/(k + rank) — so scales never have to be calibrated against each other, and a document good on both lists rises.
WITH fused AS (
SELECT id, RANK() OVER (ORDER BY s DESC) AS rank FROM (
SELECT id, BM25(items_idx.tableoid) AS s
FROM items_idx WHERE name @@ 'running'
ORDER BY s DESC LIMIT 100) lex
UNION ALL
SELECT id, RANK() OVER (ORDER BY dist) AS rank FROM (
SELECT id, emb <-> [1.0,0.0,0.0]::FLOAT[3] AS dist
FROM items_idx
ORDER BY dist LIMIT 100) vec
)
SELECT id FROM fused GROUP BY id
ORDER BY SUM(1.0 / (60 + rank)) DESC LIMIT 4;Step-by-Step: Build a Hybrid Search in 5 Minutes
Create the dictionary and the table
The dictionary decides how text is tokenized. position = true is what makes phrase queries possible later.
CREATE TEXT SEARCH DICTIONARY en (
template = 'text', locale = 'en_US.UTF-8',
case = 'lower', stemming = false,
accent = false, frequency = true, position = true
);
CREATE TABLE items (
id INTEGER PRIMARY KEY,
name VARCHAR,
emb FLOAT[3]
);Create the hybrid index
Text column, vector column and key in one statement. This is the whole of "two systems" collapsing into one line.
CREATE INDEX items_idx ON items
USING inverted (id, name en, emb ivf (metric = 'l2'));Insert the data
Index visibility is eventually consistent, so force the refresh instead of waiting for it. In production, ai_embed can produce the vectors inline at write time.
INSERT INTO items VALUES
(1, 'trail running shoe', [1.0, 0.0, 0.0]::FLOAT[3]),
(2, 'road running shoe', [0.9, 0.1, 0.0]::FLOAT[3]),
(3, 'leather dress shoe', [0.0, 1.0, 0.0]::FLOAT[3]),
(4, 'wireless earbuds', [0.0, 0.0, 1.0]::FLOAT[3]),
(5, 'marathon racer', [0.95, 0.05, 0.0]::FLOAT[3]);
VACUUM (REFRESH_TABLE) items;Filtered ANN — keyword plus vector
Shoes only, ordered by distance. Returns the two running shoes — marathon racer is closer in vector space but has no matching term, so the filter excludes it.
SELECT id, name FROM items_idx
WHERE name @@ 'shoe'
ORDER BY emb <-> [1.0, 0.0, 0.0]::FLOAT[3], id
LIMIT 2;RRF — fuse the two rankings
Now nothing is excluded: both lists rank independently and the fused score decides. marathon racer comes back — third, behind the two documents that scored on both signals.
WITH fused AS (
SELECT id, RANK() OVER (ORDER BY s DESC) AS rank FROM (
SELECT id, BM25(items_idx.tableoid) AS s
FROM items_idx WHERE name @@ 'running'
ORDER BY s DESC LIMIT 100) lex
UNION ALL
SELECT id, RANK() OVER (ORDER BY dist) AS rank FROM (
SELECT id, emb <-> [1.0,0.0,0.0]::FLOAT[3] AS dist
FROM items_idx
ORDER BY dist LIMIT 100) vec
)
SELECT id FROM fused GROUP BY id
ORDER BY SUM(1.0 / (60 + rank)) DESC LIMIT 4;Use Cases for Hybrid Search
E-commerce product search
A shopper types red running shoes. Keyword catches the exact attributes — colour, category, brand — and vector fills in the semantically similar products that use different words. One SQL statement, no external search engine, and the sales analytics that read the same table stay in the same database.
RAG pipelines for LLM apps
Documents are split into chunks, each holding its text and its embedding. Hybrid search surfaces the chunks that match on keywords and on meaning, which is what raises the quality of the context window. The langchain-serenedb package implements a VectorStore with hybrid search, and RAGFlow can use SereneDB as its doc store.
Log and observability search
Index the message text for full-text and an embedding for "find me errors that look like this one". Filtered ANN is the natural shape: an exact predicate on severity and service, then semantic ranking on the body. Works over a view, so indexing external data — Parquet in S3 — needs no ingest.
Frequently Asked Questions
Running keyword search (BM25) and vector search (ANN) over the same corpus and combining their results. Keyword brings precision on exact terms, vector brings recall on meaning, and together they cover queries that either one alone would miss.
One inverted index, built on IResearch, covers the text column and the vector column together. As a hybrid search database it combines both signals inside a single SQL query — see the hybrid search documentation for Filtered ANN and RRF.
A way to merge two ranked lists by summing 1/(k + rank) per document. It uses positions rather than scores, so BM25 relevance and vector distance never have to be calibrated onto a common scale.
No. SereneDB keeps the text and the embeddings in one index, so a hybrid search engine here needs no second datastore and no sync pipeline keeping two copies of the corpus consistent with each other.
Yes. ai_embed() calls OpenAI, Gemini, Ollama or any OpenAI-compatible endpoint straight from SQL, at write time or inline in ORDER BY. No external ETL step.
Yes. The langchain-serenedb package implements a VectorStore with hybrid search support, including HybridSearchConfig for choosing and tuning the fusion strategy.
Filtered ANN, RRF, normalized scores and weighted sum — all expressed in standard SQL rather than a config key. Which one fits depends on the workload; the hybrid search documentation writes each one out.
Get Started with SereneDB
One index, one query, five statements from empty table to fused results. The docs carry all four fusion strategies.