SereneDB has built a VectorStore integration package for LangChain, pairing flexible AI orchestration with the analytical power of a full-featured database. Three wins for your RAG pipeline:

• Unlimited analytical power — native hybrid search, metadata filtering, and relational queries let your agents retrieve on any shape and complexity of data, far beyond simple vector matching.
• Single engine - SereneDB is a full database, not just a vector store. All the surrounding data can stay in the same instance. No additional engines to maintain.
• Rapid development — the package handles all the store specifics, so you can focus on the actual application.

RAG in the modern world

Large language models are remarkable generalists with two well-known blind spots: their knowledge stops at a training cutoff, and they know nothing about your data — internal docs, tickets, codebases, product catalogs. Retrieval-Augmented Generation (RAG) is the pattern that closes both gaps. Instead of hoping the model memorized the answer, a RAG application first retrieves the most relevant pieces of your own data and hands them to the model as context, so the answer is grounded in real, current, private sources — with far fewer hallucinations and even citations to back it up.

In a few short years RAG has gone from a research idea to the default architecture for applied LLM systems: chat-with-your-docs assistants, support bots, enterprise search, code assistants — nearly every "LLM + company data" product is a RAG pipeline at its core. The mechanics have matured along the way. Documents are split into chunks and embedded into vectors, similarity search finds candidates at query time, and modern pipelines layer on hybrid keyword + semantic retrieval, metadata filtering, and re-ranking to keep quality high at scale. Which means every RAG system stands on two practical pillars: a framework to orchestrate the pipeline, and a database that can store and search vectors efficiently.

Today we will talk about one of the most powerful RAG frameworks — LangChain — and how SereneDB slots into it as the vector store.

A little about LangChain​

LangChain is an open-source framework for building applications on top of large language models. Instead of wiring raw API calls together by hand, developers compose applications from standard building blocks: a model call, a prompt template, a retriever, a memory layer. The framework's real value is in its abstractions — every provider-specific service (an LLM, an embedding API, a database) is hidden behind a common interface, so an application written against LangChain can swap OpenAI for Anthropic, or one database for another, without rewriting the surrounding Logic.

For making a RAG pipeline with LangChain one usually uses the following blocks: loader → splitter → embeddings → vector store at ingestion time, and query → embeddings → vector store (as retriever) → chat model at answer time. And here is where SereneDB appears.

SereneDB in the pipeline: the vector store

langchain-serenedb plugs SereneDB into that pipeline as the VectorStore implementation. So it fits natively into the LangChain pipeline as shown.

The integration package maps LangChain's vector store contract onto SereneDB's native capabilities: vector columns, distance operators, the IVF ANN index, BM25 full-text search, and JSON metadata.

The key point: You don't need to know SQL. Although SereneDB is a full SQL database underneath, the integration handles the entire database side internally — creating tables, declaring indexes with the right options, composing search queries, and translating filters. You work with Python objects and LangChain's standard vector store API; the package generates correct, index-aware SQL behind the scenes. Someone building a RAG application never needs to learn SereneDB's DDL or SQL query syntax to get full use of its engine — they can stay focused on the real deal: their documents, their retrieval quality, their application.

Let’s see what SereneDB brings to your RAG pipeline.

Vectors first​

Vector similarity search is the first thing you need when adding VectorStore to the pipeline. The integration stores your embeddings and queries them through SereneDB's IVF (inverted-file) ANN index for fast approximate nearest-neighbor search at scale. You choose the distance that fits your embeddings — cosine, euclidean, inner product, or manhattan — and the package keeps the index and every query aligned on it. Every standard LangChain entry point — similarity_search, similarity_search_with_score, MMR, as_retriever() — is served by that IVF index, so the whole vector-store surface is index-accelerated, not just a special "fast path."

What about metadata?​

Vector search alone gives you the semantically closest chunks, but "closest" isn't always "relevant" — you often want the nearest neighbors among last quarter's tickets, from this author, in the pricing category. In the real world documents carry that structure by metadata — categories, timestamps, authors, tags. And constraining the similarity search by it is what turns a decent result set into the right one. The store gives you a rich filter language rather than bare equality: comparisons and ranges, set membership, existence and text patterns. Every filter compiles to SQL predicates evaluated inside the engine, right beside the vector scan, never as a slow post-filter in Python. Metadata can sit in a zero-ceremony JSON column or be promoted to typed columns when you know the schema — so filters ride an index scan instead of a full table scan.

Content is not just payload​

So far the content has been along for the ride — embedded into a vector, filtered by its metadata, then handed back on a hit. But the text itself is a search signal too, and SereneDB can search in it directly. Turn on hybrid search and the store builds a combined index that fuses semantic vector similarity with SereneDB's native BM25 full-text ranking in a single query, so exact terms that embeddings miss — product codes, function names, rare keywords — still surface, combined by the fusion strategy you choose. And because the content column is inverted-indexed, the same machinery lets you bring full-text operators too.

You own your data​

SereneDB is self-hostable — and for RAG that is more than an ops preference. The documents you index are usually exactly the data you can't send to a third party: internal knowledge bases, contracts, source code, customer records. Run SereneDB on your own infrastructure and the entire retrieval side of the pipeline — documents, embeddings, queries — stays inside your perimeter. Your data stays yours.

Not just VectorStore​

VectorStore is only one corner of the system. SereneDB is a full database rather than a dedicated vector engine, so the same instance that serves your embeddings also serves everything around them: the relational tables your application already needs, analytical queries over your document corpus, full-text search on its own, JSON Storage — one connection string, one operational surface, standard PostgreSQL tooling. Your chunks can live next to the users, sessions, and business data they belong to. While this part is not covered by this LangChain integration package - you can use SQL to access the data and run additional tasks on it. The SereneDB documentation covers the full feature set well beyond what this integration touches.

And what matters the most is that all features are in one engine: IVF-accelerated vector similarity, BM25 full-text ranking fused with it in a single hybrid query, and inverted-index-backed metadata filtering that runs next to the vector search instead of after it. You get the retrieval quality tricks of a modern pipeline — semantic recall, exact-term matching, precise filtering — without stitching together a vector engine, a search engine, and a filter layer from three different products.

Getting started​

Now it is time to see SereneDB in action. As a tradition - a small HelloWorld example. You will need at least python 3.10 to run the example.

First of all we get a fresh SereneDB instance running. In this example we will use convenient script that will handle the details of creating a docker container:

curl https://install.serenedb.com | sh

Alternatively you can use other means for creating the instance - look at our QuickStart guide

Now the langchain-serenedb integration package

pip install langchain_serenedb

For the purposes of this example we will use langchain core provided fake embedder. But it is trivial to replace it with any of the embedder integrations supported by LangChain.

from langchain_serenedb import SereneDBEngine, SereneDBVectorStore, IVFIndex
from langchain_core.embeddings import DeterministicFakeEmbedding
from langchain_core.documents import Document

# This is the embedder. Replace it with the real one.
my_embeddings = DeterministicFakeEmbedding(size=768)

# Connect in a postgres style
engine = SereneDBEngine.from_connection_string(
    "host=127.0.0.1 port=7890 user=postgres dbname=postgres"
)

# Create the table and its IVF ANN index in one call.
# All internal SQL is generated automatically.
engine.init_vectorstore_table("my_docs", vector_size=768, vector_index=IVFIndex())

# Spin up the VectorStore
store = SereneDBVectorStore.create_sync(
    engine, embedding_service=my_embeddings, table_name="my_docs"
)

# A tiny corpus to index (page content + optional metadata)
docs = [
    Document(page_content="SereneDB speaks the PostgreSQL wire protocol.", metadata={"topic": "intro"}),
    Document(page_content="The IVF index accelerates nearest-neighbor search.", metadata={"topic": "index"}),
    Document(page_content="Hybrid search fuses BM25 keyword ranking with vectors.", metadata={"topic": "search"}),
    Document(page_content="Metadata filters run inside the engine, next to the vectors.", metadata={"topic": "filter"}),
    Document(page_content="Set sdb_nprobe to trade recall for latency per query.", metadata={"topic": "tuning"}),
]

# Add some document to the index
store.add_documents(docs)

# And run your search!
results = store.similarity_search("how do I configure the index?")

From here it behaves like any LangChain vector store — similarity_search, similarity_search_with_score, max_marginal_relevance_search, get_by_ids, delete, and as_retriever() for dropping it into a chain. Of course this is only the basic setup, we will cover more advanced features in the following chapter.

Advanced Topics

Here we provide a short list of most usable features in detail. And dive into some internal machinery that powers our VectoreStore.

Features walkthrough​

Accelerated vector search with IVF. SereneDB's ANN index is an IVF (inverted file) index, and the integration exposes it as a plain Python object: IVFIndex. Declare one and the package derives the distance metric from your chosen DistanceStrategy (Euclidean/L2, cosine, inner product) and guarantees the index and every query it issues agree on it — a consistency detail that trips people up when managing ANN indexes by hand, handled here automatically. Optional quantization trades a little recall for a lot of memory and speed: sq8, sq4, pq (with configurable sub-quantizers), or rabitq (1–9 bits) — each just a constructor argument. For bulk loads, creating the index after loading lets SereneDB train better IVF clusters; for incremental workloads, creating it with the table (init_vectorstore_table(..., vector_index=IVFIndex())) means search is accelerated from the first row. apply_vector_index(), reindex(), and drop_vector_index() manage the index on a live store — no DDL statements to write or migrations to maintain.

Tunable query-time recall. IVFQueryOptions controls the recall/latency trade-off per query: nprobe sets how many IVF cluster lists are scanned (applied as the sdb_nprobe session setting), and rerank_factor sizes the exact-distance rerank pool when a quantized index is in play.

Hybrid search: BM25 + vectors in one query. Semantic search is great at paraphrase but can miss exact terms — product codes, function names, rare keywords. Build the combined full-text + vector index with a HybridIndexConfig, give the store a HybridSearchConfig, and it runs a single query that fuses SereneDB's native BM25 ranking with vector distance. Three FusionStrategy options are available:

• RRF (Reciprocal Rank Fusion, the default) — combines ranks rather than scores (sum(1 / (rrf_k + rank))), so BM25 scores and vector distances fuse without any normalization.
• Weighted — min-max normalizes each branch (inverting distance so nearer = higher) and takes a weighted sum, preserving score magnitudes.
• Weighted sum — a plain weighted sum of the raw branch scores. See more detailed description with examples at our documentation page

Metadata filtering. Searches accept a MongoDB-style filter dictionary — $eq, $ne, $lt/$lte/$gt/$gte, $in/$nin, $between, $exists, $like/$ilike, composed with $and/$or/$not. On inverted-indexed columns you also get full-text operators — $startswith, $regex, $fuzzy, $ngram, $match, $phrase — for prefix, regex, typo-tolerant and phrase matching. The package translates the dictionary into database predicates for you, so filtering runs inside the engine, right next to the vector search — no post-filtering in Python, and no query syntax to learn. Flexible metadata storage — and indexes for it. Metadata can live in a single JSON column (zero schema ceremony) or be promoted to explicit typed columns via Column/ColumnDict when you know your schema. Either way it can be indexed for fast filtering: MetadataColumnIndex for dedicated columns and JsonFieldIndex for individual fields inside the JSON blob, bundled through MetadataIndexConfig. Indexing a field inside a JSON document correctly has subtle expression-matching rules at the database level; the package encodes them so the index is always actually used — another piece of database expertise you don't have to bring yourself.

MMR retrieval. max_marginal_relevance_search is supported for result diversification — useful when the top-k nearest neighbors are near-duplicates and the chain benefits from broader coverage.

Sync and async, one behavior. Every operation exists in both flavors — add_documents/aadd_documents, similarity_search/asimilarity_search, and so on — with the sync store delegating to the same async core, so behavior never drifts between the two.

How it works​

Under LangChain's tidy add_documents / similarity_search surface, the package maps a collection onto a deliberately simple SereneDB layout: one table and one inverted index. Its real job is to keep the DDL it writes and every query it later issues in lockstep — same distance metric, same extraction expressions, same dictionary — so searches actually hit the index instead of quietly degrading into a full scan.

The table. init_vectorstore_table creates a single table per collection: an id primary key, a content TEXT column, the embedding as a fixed-size FLOAT[N] array, and your metadata. Metadata has two homes, and you can mix them: a catch-all JSON column (the default — no schema to declare) or explicit typed columns promoted with Column when you know a field's shape and want to index or constrain it.

from langchain_serenedb import Column

engine.init_vectorstore_table(
    "my_docs", vector_size=768,
    metadata_columns=[
        Column("category", "TEXT", nullable=False),
        Column("year", "INTEGER"),
    ],
)

Anything you don't promote still round-trips through the JSON column, so promoting a field is an optimization, never a requirement.

One index, several columns. SereneDB's inverted index is not vector-only: a single USING inverted (...) can carry the embedding column (with the ivf operator class for ANN), the content column (analyzed for BM25), and any number of metadata columns and JSON sub-fields — all at once. The package builds exactly one such index per collection and routes every search through it. That is the whole reason a hybrid, metadata-filtered query can be one SQL statement: the vector ranking, the keyword ranking, and the filter predicates are all served by the same index. And it is all powered by our own IResearch library that was started in 2016, and we have never stopped improving it. Check our benchmarks against other search engines in Search Benchmark Game to see the actual performance results.

The full-text dictionary. Full-text scoring needs a text-search dictionary — the analyzer that turns raw content into scored tokens. The configuration splits cleanly by concern: a build-time HybridIndexConfig describes the dictionary (used when the index is created — the package issues a CREATE TEXT SEARCH DICTIONARY and references it on the content column), while a query-time HybridSearchConfig carries the fusion knobs (used on every search). The default dictionary is deliberately universal: a segmentation template, lower-cased, with frequency, position, and norm enabled — each one unlocking a capability, frequencies for BM25 scoring, positions for phrase and n-gram matching, norms for the language-model scorers. The dictionary is created in the table's own schema so an index living outside public can still resolve it — a subtlety the package handles so you never meet it. Both sides are overridable:

from langchain_serenedb import HybridIndexConfig, HybridSearchConfig, FusionStrategy

# build-time: how the content column is analyzed
index_cfg = HybridIndexConfig(
    dictionary_options="template = 'segmentation', case = 'lower', "
                       "frequency = true, position = true, norm = true",
)
# query-time: how the lexical and vector rankings fuse
search_cfg = HybridSearchConfig(fusion=FusionStrategy.RRF, scorer="BM25")

Deciding what to index. By default the package indexes every declared metadata column verbatim — one token per value — which is exactly what lets a plain =, IN, or range filter be answered from the index scan instead of a row-by-row recheck. There is a safety net: in this index-everything mode a column whose type the inverted index cannot take verbatim (a NUMERIC, a UUID, an INTERVAL) is quietly skipped, so the automatic index can never fail to build. When you want control, MetadataIndexConfig states precisely what joins the index — a subset of columns, a JSON sub-field, or a column analyzed for full-text.

from langchain_serenedb import (
    MetadataIndexConfig, MetadataColumnIndex, JsonFieldIndex,
)

metadata_index = MetadataIndexConfig(
    columns=[
        MetadataColumnIndex("category"),                                # verbatim → =, IN, range
        MetadataColumnIndex("title", dictionary="langchain_fts_dict"),  # full-text
    ],
    json_fields=[JsonFieldIndex("attrs.brand", "TEXT")],                # a field inside the JSON 
)

Here is a schematical data flow inside SereneDB database:

Two choices there carry real database expertise. Attaching a dictionary to a column flips it from verbatim to full-text-analyzed: you gain the $regex / $phrase / $fuzzy operators on it but give up plain-equality pushdown (analyzed columns match through @@, not =). And indexing a field inside the JSON blob only works if the index expression and the query expression are byte-identical — same ->> arrow, same ::type cast on both sides — or the filter pushes down but returns the wrong rows. The package generates both sides from one definition so they cannot drift.

Why can't the metric drift? The same principle governs the vector side. An IVF index only accelerates a query whose distance operator matches the metric the index was built with; let them fall out of step and the search silently reverts to a full scan. The package derives both the index metric and the query operator from a single DistanceStrategy, so the guarantee holds by construction — the recurring theme of the whole integration: you declare intent in Python, and the package keeps the generated SQL self-consistent.

Recap

By bringing vector similarity, BM25 full-text ranking, and metadata filtering together into a single, SQL-backed engine, SereneDB is a good fit for the architecture of modern LangChain RAG pipelines. Instead of managing multiple disparate systems, you get a cohesive retrieval layer that handles complex data shapes with index-accelerated performance. We invite you to try the langchain-serenedb package or dive into our documentation to see how SereneDB can streamline your own AI applications.

Additional links:

• https://pypi.org/project/langchain-serenedb
• https://docs.serenedb.com
• https://github.com/serenedb/serenedb
• https://github.com/serenedb/langchain-serenedb

Welcome back to The State of Serene, our monthly note on what shipped and where SereneDB is heading.

August went into search. New features: sloppy phrase search, a Lucene query parser that reads queries the way Lucene reads them, REINDEX for remote indexes that keep themselves up-to-date, per-branch scoring control, an idf() scorer and Azure blob storage. Plenty of performance work too: the postings read path got rewritten and runs up to 2.9x quicker across 1076 queries, BM25 score computation nearly halved and quant = 'none' vector search became a real quantizer instead of re-reading raw vectors. The catalog also split away from the data, which is what makes index recovery parallel. Outside the repo, SereneDB became a selectable doc store in RAGFlow. And we pointed the whole thing at Dota 2 and shipped Serene Why You Lost.

v26.08.2 is the latest release.

What happened in August​

Sloppy phrase search​

An exact phrase misses the text that drops a word in the middle. "zion machine" does not find "zion sent a machine". Until August the way to ask for that was to build a disjunction of every gap variant by hand.

Slop is a budget for the whole phrase instead:

SELECT id FROM docs_idx WHERE description @@ ts_phrase('zion machine', slop := 3);
SELECT id FROM docs_idx WHERE description @@ ts_phrase('group children')::slop(1);
SELECT id FROM docs_idx WHERE b @@ to_tsquery('"quick fox"~1');

Three spellings for the same thing. slop := sets it on the call, ::slop(N) applies it to a phrase that already exists, including one from phraseto_tsquery. Lucene's "..."~N reaches it through to_tsquery. Declared gaps still work: with ts_phrase('group', 1, 'children', slop := 0) the budget counts deviation from the declared gap rather than from adjacency.

Docs: proximity search with slop and ts_phrase.

Lucene query syntax​

to_tsquery takes a Lucene or Elasticsearch query_string expression. The parser behind it got rebuilt to read one the way Lucene does. What it reads:

Terms are whatever the syntax has not claimed, so café, u.s.a and rock-n-roll are terms. A hyphen or a plus inside a word belongs to the word. A term goes through the analyzer. A prefix, wildcard, fuzziness or range bound gets normalized instead of tokenized, which is the distinction that keeps ["alpha beta" TO gamma] searching from alpha beta rather than from alpha.

Anything SereneDB has no algebra for is refused by name instead of quietly ignored. A query that cannot be read says so instead of answering zero.

Docs: to_tsquery.

Search got faster​

The postings read path, the term-iterator contract and score pruning all got rewritten. Measured over the 1076 queries of search-benchmark-game against the tree before the rewrite:

Computing a BM25 score came out nearly twice as fast on its own.

Where it comes from: two new block encodings that carry no payload at all, so a run of consecutive doc ids decodes from its header and folds into a consumer's mask in one operation. Block consumption moved out of the iterator and into the format layer, so the block disjunction and the deleted-docs mask share one implementation instead of each open-coding it. The in-block bit scatter runs 8 interleaved chains, because consecutive doc ids usually land in the same 64-bit word and serialise the loop on store-to-load forwarding. In-block seek is a branchless fixed-width binary search, so there is no data-dependent branch left to mispredict.

Search over a lakehouse, without a second copy of your data​

Your data in your own bucket stays the source of truth, whether that is an Iceberg table or a directory of Parquet, CSV or JSON. Any engine writes to it, Spark or BigQuery or Flink or SereneDB itself. SereneDB holds only the derived index on local disk: term dictionaries, vector structures, the columns you chose to index. There is no second copy of the corpus, so losing the node means rebuilding an index rather than restoring a database. Columns you did not index are still selectable, materialized from the source for the rows a query actually matched.

The piece missing until August was the barrier between writing and querying.

REINDEX INDEX chunks_idx;

CREATE INDEX rpr_idx ON rpr_v USING inverted(id, body rpr_en)
  WITH (reindex_interval = 100);

One pass compares the source's current committed state against what the index holds, applies the difference and publishes atomically, so readers see the previous complete state or the new one and never a partial index. When it returns, everything committed before it is searchable. That turns a pipeline into three steps: writers commit, one REINDEX, consumers start. reindex_interval runs the pass on a schedule instead. REINDEX INDEX CONCURRENTLY is accepted too, because the pass never blocks readers either way.

How much work a pass does depends on what it is reading:

For a catalog-attached Iceberg table a pass forces a fresh table load even inside the server's staleness window, which is what makes it a barrier rather than a hint.

The whole thing from an empty catalog, including the hybrid queries at the end, is in Search over Iceberg. Reference for the refresh itself: refreshing the index.

Control how a score is evaluated​

Scoring a boolean query is two decisions: how each branch scores and how the branch scores combine. Both are overridable per node now.

::merge(...) sets the combine policy. The default is sum, so a document matching fox and cat gets both contributions added. max takes the best branch instead:

SELECT id, BM25(idx.tableoid) FROM idx
WHERE (body @@ 'fox' OR body @@ 'cat')::merge('max');

It binds to its own node, so nesting means something. max on an inner node still lets the outer node add dog on top. max on the outer node takes the best of all three.

::score(...) overrides the scorer for one subtree while the rest of the query stays on the ORDER BY scorer:

SELECT id, BM25(idx.tableoid) FROM idx
WHERE body @@ 'fox' OR (body @@ 'cat')::score('constant(1)');

::score(NULL) takes a subtree out of scoring entirely. It still selects rows, contributes nothing and asks for no index features, so its postings are read without frequency or norms. That is the shape Elasticsearch spells as a filter clause.

The two compose. A per-branch scorer under a group policy competes on its own terms, so a constant(10) branch survives a max against a BM25 sibling. An unscored branch has nothing to compare, so the other branch wins by default.

max earns its keep on synonyms. Expand car into car OR automobile and the default sum gives a document that happens to use both words two contributions. It then outranks a document that is just as relevant and picked one spelling. Those branches are one concept and max scores them as one.

Both ::score and ::merge ride on the TSQUERY value rather than on the predicate, the same family as ::boost and ::slop, so they reach a group built by ts_any or ts_all as readily as a parenthesised OR:

SELECT id, BM25(idx.tableoid) FROM idx
WHERE body @@ ts_any(['car', 'automobile'])::merge('max');

Riding on the value is also why they survive a prepared statement. WHERE b @@ $1::score('constant(42)') makes the parameter that type, so the modifier arrives with the bound value rather than being lost before the index sees it.

There is also a new idf() scorer: a document scores the inverse document frequency of the matched term alone, with no term frequency and no length normalisation. It works anywhere a scorer works, optimize_top_k = 'idf()' included. It scores columns indexed without a dictionary too. bm25(k1 = 0) is the same formula and used to return zero, which is fixed as well.

Docs: scoring.

Azure, plus credentials for every cloud​

az:// URIs work now in COPY, read_csv, read_parquet and glob, with CREATE SECRET (TYPE azure) for connection strings and scoping. You can build an inverted index over an az:// Parquet or CSV view the same way you would over S3.

Iceberg catalogs got Google service-account auth: CREATE SECRET (TYPE ICEBERG, PROVIDER google) with service-account-key and GCE metadata-server modes, so BigLake catalog tokens renew themselves under a machine identity with no user credentials involved. Verified against a real BigLake catalog with a forced token expiry mid-session.

Object-storage reads got about a fifth quicker in the same stretch: large Azure reads 20% warm and 21% cold, large HTTPS reads 21%.

Docs: AWS, Azure and Google Cloud credentials, plus Iceberg catalog authentication and the BigLake cookbook.

pgstream works against SereneDB, plus a wider Postgres surface​

Point a Postgres logical-replication client at SereneDB and it works now. pgstream is the one we tested with. Three gaps each stopped it cold and silently, all three in how a pg catalog answered a question. All three turned up by diffing its wire capture against the same client aimed at real Postgres.

pg_proc and pg_aggregate are populated. pg_proc listed only macros created in the current database, so 3461 functions existed and exactly one was visible. \df, ORM introspection and information_schema.routines and parameters all saw nothing. Both are filled from a walk over the system catalog now.

Vector search: Panorama pruning and a cheaper descent​

quant = 'none' used to mean no payload at all, so every query re-read raw vectors out of the columnstore to rank them. It is a real quantizer now, built on faiss's Panorama layout: a PCA-rotated basis with per-level suffix norms, which lets the posting scan drop a candidate on an exact Cauchy-Schwarz bound against the running k-th best rather than reading it in full. Same answers, less reading.

Radius queries go through the payload for the same reason, with a constant pruning threshold instead of a raw reranker over the columnstore. Under a lossy quantizer a survivor's reported distance is rescored exactly from the index's own vectors, so the number you get back is the real one.

The quantizer writer and reader protocol is a single block contract now, which is what lets Panorama's batched layout sit alongside PQ and RaBitQ fast-scan groups and flat scalar-quantizer records without each one growing its own hooks.

Then the descent itself got cheaper. Probing N clusters on a deep centroid tree meant expanding every child at every level. Most of that work never reaches a cluster you scan. sdb_ivf_max_search_fanout caps how many children a node expands, defaulting to 16, which decouples descent width from how many clusters you asked for. It raises itself when the cap is too small to supply the requested clusters, so it cannot starve a query. The nprobe setting is now sdb_ivf_search_nprobe. sdb_rerank_factor takes fractional values.

SereneDB is a doc store in RAGFlow now​

RAGFlow is an open-source RAG engine with around 90k stars. Since August SereneDB is one of the document stores you can put behind it, on both the Go and the Python path. DOC_ENGINE=serenedb and it runs. The integration is not ours. Our fellow community member deadtrickster wrote it: 26 files and about 3,900 lines. It merged upstream on 4 August and ships in RAGFlow v0.27.0 onward.

Why it fits is the thing this blog keeps saying. One inverted index carries the scored text column and the IVF vector column, so hybrid retrieval is one SQL statement instead of two systems and a merge step. The connector puts one table per tenant with kb_id as a filter column, which is the Elasticsearch layout rather than a table per dataset, so BM25 statistics stay computed over the whole tenant corpus instead of per knowledge base.

Their numbers, measured on the Python path over a 247,665-chunk multilingual corpus with weighted fusion, against Elasticsearch on the same gold set:

Same MRR, a hair behind on mean rank, roughly 5x quicker. At concurrency 16 it did 514 QPS against Elasticsearch's 241, with a p99 of 42 ms against 108 ms. Building the index over those 247k chunks took 5.6 seconds. Migrating all 247,665 documents through the connector produced zero errors.

Those are the contributor's measurements on their corpus rather than ours. We have not reproduced them. We are quoting them because the methodology is in the PR.

Why You Lost, an analytics showcase built on SereneDB​

Serene Why You Lost takes a finished Dota 2 match, lets you pick which of the ten players is you and tells you what actually cost you the game.

It exists to show what search plus analytics in one engine does to a problem that is not logs. A match is a long event stream with ten people deciding at once and a one-word result hiding all of it. Finding the moments that mattered is a search problem. Judging whether a decision was good is an analytics question over thousands of similar situations. Both run in the same queries, which is the whole argument this database makes, applied to something you can check against your own memory of the game.

A scoreboard says you died six times and finished an item late. It cannot say which of those mattered, what led to it or what the alternative was. Why You Lost answers at three depths. The overview accounts for how the game developed and where momentum shifted. The deep analysis finds the moments that moved your chances of winning and explains why they moved them. The interactive 2D replay puts those moments back in context, with every fight on one timeline and positions, movement and health second by second.

No language model writes the advice. Every comparison comes from real match data and it prints the sample size next to the number, so you can decide for yourself whether to believe it.

It went up as a seven-day test during The International 2026 and it's still online. Whether it stays depends on whether people use it, so go break it. The writeup has the rest.

Search-backed tables take indexes​

A search-backed table is the iresearch columnstore with, as of August, optional indexes on top. WITH (storage = 'search') and the rows go straight into iresearch segments. It is not transactional. What you get back for that is speed.

The indexes are the new half. Until August this was a table you could not index at all, which limited it to whatever a sequential scan could answer. Now it carries inverted indexes with the same syntax a regular table uses. Each indexed field gets a virtual field id that maps back to the real column on materialization, so one column can be indexed several times with different tokenizers while sharing a single store.

The bulk-load path also stopped writing WAL chunk files. It flushes and fsyncs whole index segments and records the segment name in the WAL instead, so recovery adopts the segment when its tick committed and background cleanup wipes it when it did not.

Plain inserts are at parity now. On 10M rows of ClickBench a search-backed table does insert plus commit in 4.89 s where a transactional one takes 5.20 s. It used to be around 30% slower than that. Give both tables inverted indexes on WatchID and URL and it stops being a comparison:

Two limits worth knowing before you reach for it. There is no backfill yet, so indexes have to be added while the table is empty. Only one IVF index per column is allowed. Both are follow-ups rather than design decisions.

Also shipped​

• DDL is transactional now. A CREATE or DROP that fails no longer leaves the catalog half-changed.
• Recovery of inverted indexes runs in parallel and is no longer bound by how much memory the machine has.
• DML against an inverted index runs in parallel too.
• Sequences got faster.
• Fuzzy expansion has its own cap, sdb_levenshtein_max_terms, default 64. It used to take its limit from sdb_scored_terms_limit, which is documented as a scoring-cost knob, so setting that knob changed result sets: over 2001 terms within edit distance 2 of cat, SET sdb_scored_terms_limit = 50 returned exactly 50 rows.

What to expect in September​

• Logical replication subscriptions and triggers. CREATE SUBSCRIPTION so an attached Postgres streams its changes in rather than being re-read. August made SereneDB usable as a replication target; this is the other half.
• Row-level security. Policies per table, on top of July's roles and grants.
• Another round of search performance. The postings work in this issue is not finished. The next round is aiming at 1.5x to 2x.
• Geospatial. A spatial type and index surface.
• Even faster search. We've almost finished a huge rework of our search execution.
• More benchmarks. Still open from last month. Vector benchmarks are the first ones we owe you.

Kudos​

Plenty of this month came from outside the core team.

aksel2904 built sloppy phrase search, both the engine side and the SQL surface. He also contributed the regexp filter we shipped back in June, which makes two features of the search surface now.

deadtrickster did awesome work. Apart from the RAGFlow integration, he also built the tool we should probably have written ourselves. serenedash is a live terminal dashboard for a SereneDB server: storage and the spill split, pool memory against RSS and swap, sessions and how far along their statements are, per-thread CPU, a perf-backed profile and whether index maintenance is keeping up. The same collectors are exposed over MCP, so an agent can read the live server instead of having panels pasted at it. Credentials are optional. A panel that genuinely needs the server says which of "no driver", "no credentials" or "cannot connect" applies rather than drawing a zero and calling it a reading.

On the activity view, one keypress plans a statement that is already running. EXPLAIN has always been something you could type, but aiming it at the query currently burning a core means fetching its text out of pg_stat_activity and quoting it back. On that deployment the text was 68 KB. EXPLAIN does not execute, so it is safe to point at something hung.

He also filed several of the bugs we fixed this month, found by running SereneDB behind a real workload and reporting exactly what broke. Thanks as well to emarsden, who keeps holding the SQL surface to the standard and to deymon-d, who fixed phrase with intervals.

And thank you to everyone with work still in progress: seb-06, w3lld1, ivan-digital, romanpovol and afigor2701.

Want to be in the next one? We tag beginner-friendly work with good first issue, so grab one, ask questions in the issue and we'll get you going.

v26.08.2 is the current release. Grab it and point it at something. If you'd rather look before you install, the code search demo is live and ⌘K on the docs gets you SereneDB searching SereneDB's documentation. Every raw benchmark result is at serenedb.com/searchbench.

If you like what you see, ⭐ star us on GitHub. Hit a rough edge? Open an issue. The Kudos section above is people who did exactly that.

See you in the next State of Serene.

We build a database, which means we spend a lot of time pointing it at datasets and asking increasingly difficult questions. Usually that ends in a benchmark. This time we pointed it at Dota 2 instead.

The result is Serene WYL, short for Why You Lost: a tool that takes a finished match and explains what happened from one player's point of view.

A scoreboard can tell you that you died six times, finished an item late or lost the last fight. It cannot tell you which of those things actually mattered, what led to them or what you could have done differently. WYL is built for that second set of questions.

Paste a match link, choose which of the ten players is you and start with the big picture. From there you can go as deep as you want.

From the match overview to individual decisions​

Serene WYL presents the same match at several levels.

Move from the match overview into draft, fights and build analysis.

The general analysis gives you a quick account of the game: how it developed, where momentum shifted and which parts of the match deserve a closer look. It is the place to start when you want to understand the game without watching the whole replay again.

The deep analysis looks at the decisions behind the result. It finds moments that changed your chances of winning, explains why they mattered and gives you a more useful answer than a list of end-of-match statistics. You can inspect the parts that matter to you and skip the rest.

The interactive 2D replay connects the analysis back to the match itself. Fights are collected in one timeline, and you can jump directly to any of them to see positions, movement and health change second by second. It is a compact way to reconstruct a game while keeping the important moments in context.

Jump between fights and inspect the match on an interactive 2D map.

WYL also reviews picks, builds and other match decisions in the context in which they were made. The goal is not to hand out universal rules. It is to show what made sense in this particular game and offer alternatives for similar situations.

How it works​

The flow is deliberately simple:

1. Paste a link to a finished match.
2. Select your player.
3. Read the overview, open the deeper analysis and jump into the replay wherever you want more context.

The conclusions come from the events in the match and patterns across real games. This is not an AI coach generating plausible-sounding advice. Every comparison is grounded in match data, and WYL shows the sample size next to the numbers so you can judge the evidence for yourself.

That matters because Dota is situational. A decision can be good in one match and terrible in another depending on the draft, timing, map and nine other players. Looking at one isolated number removes that context. Looking at many similar situations puts some of it back.

Why we built it​

SereneDB combines search and analytics in one database. A Dota match is a fun way to show what that means: a long stream of events, ten players making decisions at once and a simple result hiding a complicated story.

Serene WYL turns that stream into something you can explore at different depths. The overview summarizes the match, the deeper layers find the moments worth examining and the replay lets you return to the original context. It is the same basic problem we work on with other kinds of data: find what matters, measure it and make the result understandable.

The current version is a draft, and we already have many more metrics we want to add. We are running it in test mode for seven days during The International 2026. If players find it useful, we will keep it online longer and turn it into a deeper analytics platform.

Try Serene WYL, then tell us what it got right, what it missed and which analysis you want next. If you want to help us keep building experiments like this, star SereneDB on GitHub.

Ask a team where their text lives and you usually get two answers. The data of record sits in ClickHouse or Snowflake or a few terabytes of Parquet on S3. The searchable copy sits in Elasticsearch. Between them runs a pipeline that somebody wrote two years ago, somebody else maintains now and everyone has opinions about.

The duplication is the obvious cost. Every searchable row exists twice and the search copy tends to be the expensive one, because a document store keeps the original JSON alongside the index so it can hand it back to you.

The subtler cost is that the two halves can't talk to each other. Elasticsearch will rank your documents beautifully and then give you a list of ids. If your next question is "and what was the weekly revenue on those?", you write application code to stitch two result sets together. Usually you just don't ask.

SereneDB exists to collapse that arrangement into one engine: Elasticsearch-grade search and ClickHouse-grade analytics, both addressed over the Postgres wire protocol. This release extends it in a direction we've been working toward for a while. You can now build a full-text index over a table that belongs to a different database entirely.

What "search" means here​

Lots of analytical databases have bolted on a text index and put "full-text search" on the feature list. You find out how deep it goes the first time a product manager asks for typo tolerance or ranking that goes beyond recency.

Ours comes from IResearch, a C++ information retrieval library our team has been building since 2016. It's what we benchmark against Lucene and Tantivy in Search Benchmark, The Game. SearchBench puts it against Elastic, OpenSearch, ParadeDB and ArangoDB. In SQL you get:

• BM25 ranking with real term frequencies and document norms
• phrase search and sloppy phrase search with a slop window
• fuzzy terms, prefixes and wildcards
• facets and aggregations computed over the matched set
• geospatial predicates on points and geometries
• vector search: IVF with product quantization, RaBitQ, and 4- and 8-bit scalar quantization
• hybrid ranking that fuses text and vector results with RRF

Because one planner sees the whole query, a search predicate and an analytical rollup are the same statement:

SELECT date_trunc('day', ts) AS day,
       count(*)              AS hits,
       avg(rating)           AS avg_rating
FROM reviews_fts
WHERE body @@ ts_phrase('battery life') AND lang = 'en'
GROUP BY day
ORDER BY day;

In the two-system world that's a search request, a JSON response, a list of ids sent back to your app and a warehouse query with a 40,000-element IN clause. Assuming the id set is small enough to ship at all.

Where an index can point​

Here's the design decision the rest of this post depends on. In SereneDB an inverted index does not belong to one table. It asks a data source for two things: a way to read the rows once at build time and a way to get a given row back later if a query needs its contents. Anything that can answer both can be indexed.

So CREATE INDEX ... USING inverted(...) currently accepts:

The middle rows are what we've been calling zero-ETL remote search: BM25 and vector search straight over a data lake with no ingestion job in the middle. We wrote that up for source code in code search over a data lake.

A live database behaves differently from a Parquet file. It has its own primary key, its own query planner and its own preferred way of being asked for two thousand rows.

Connecting a database​

SereneDB speaks the Postgres wire protocol, so it borrows Postgres's vocabulary for talking to other databases. Two foreign data wrappers ship today, postgres_fdw and clickhouse_fdw. Nothing above the connector layer is specific to either one, so adding an engine takes a connector and nothing else. Two generic wrappers are in progress right now, ODBC and ADBC. Between them they reach most of the database market. More on those at the end.

CREATE SERVER events FOREIGN DATA WRAPPER clickhouse_fdw
  OPTIONS (host 'ch.internal', port '9000', database 'prod');

GRANT USAGE ON FOREIGN SERVER events TO analysts;

A CREATE SERVER is catalog DDL. It survives restarts, re-attaches on boot, shows up in pg_foreign_server and participates in dependency tracking like any other catalog object. An ATTACH lasts only as long as the session that made it. Credentials live on the server definition (one shared connection identity, the way ClickHouse itself does it) and access is a grant.

A foreign server survives a restart. The server definitions and the inverted indexes live in SereneDB; the tables stay in the databases you already run, reachable over each engine's own protocol. Restart SereneDB and the servers re-attach on boot.

For ClickHouse the connector talks the native TCP protocol, columnar blocks and LZ4, over a pooled and health-checked connection layer, with projection, filter and ORDER BY / LIMIT pushdown, cardinality from system.tables and automatic rebinding when the remote schema shifts under it. After that the remote table is just a table:

SELECT count(*) FROM events.prod.hits;

Indexing it​

Everything in this section runs as written. It points at ClickHouse's public playground, so there is nothing to set up on the ClickHouse side at all. Two commands and a paste. The only prerequisite is Docker.

# start SereneDB locally (brings up SereneUI too)
curl -fsSL https://install.serenedb.com | sh

# connect
psql -h localhost -p 7890 -U postgres

The installer prints the exact connect and teardown commands when it finishes, along with a browser URL for SereneUI — use those if it landed on a different port. If there is no psql on the machine it ships one too. Then paste this at the prompt:

CREATE SERVER play FOREIGN DATA WRAPPER clickhouse_fdw
  OPTIONS (host 'play.clickhouse.com', port '9440', database 'default',
           user 'play', secure 'true');

CREATE TEXT SEARCH DICTIONARY en (template = 'segmentation', case = 'lower');

CREATE VIEW hn AS
  SELECT id, title, "by", score, url
  FROM play.default.hackernews
  WHERE type = 'story' AND score > 500
  LIMIT 1024;

CREATE INDEX hn_fts ON hn USING inverted(id, title en)
  INCLUDE ("by");

SELECT title, "by", score
FROM hn_fts
WHERE title @@ ts_phrase('rust')
ORDER BY score DESC
LIMIT 4;

                      title                       |      by       | score
--------------------------------------------------+---------------+-------
 Async-await on stable Rust                       | pietroalbini  |  1102
 I have written a JVM in Rust                     | lukastyrychtr |   718
 Tauri 1.0 – Electron Alternative Powered by Rust | Uninen        |   715
 Rust/WinRT Public Preview                        | steveklabnik  |   628

The index build takes about two tenths of a second, the search about the same. Three columns end up in three places. title is tokenized into the index, "by" is included so it comes off local disk and score is neither — it is fetched back from ClickHouse by id, but only for the four rows that survived the LIMIT. ClickHouse has no idea any of this happened.

The LIMIT 1024 is there because this runs on somebody else's server. The playground is public and rate-limited, so the example takes a small slice and stays well inside the quota — enough to watch the thing work and no setup on your side at all. Point it at your own ClickHouse the moment you want more than a look. The slice is arbitrary, so your four titles will not be these four.

One more, because it costs people an afternoon: the dictionary above sets case = 'lower'. The default is none, which indexes terms exactly as written and then ts_phrase('rust') matches nothing at all while ts_phrase('Rust') matches everything.

How much of your data the index keeps​

This is the knob people usually don't know they have.

By default the index holds postings, positions, norms and document lengths, plus whatever key it needs to find a row again. Your actual column values aren't in there. Ask for them and SereneDB goes and reads them from the source. INCLUDE changes that per column: named columns get written into the index's own columnstore and reading them afterwards costs one local read and no trip to the source at all.

Included columns stay local. id and the BM25 score come out of the index itself, author and posted_at out of the index's columnstore because they were included. Only text is fetched from ClickHouse, by primary key, for the ten rows that survived the LIMIT.

So you get a dial.

Store nothing. Smallest possible index. Good when the source is fast and local or when most queries only need counts, scores and ids anyway. A surprising number of "search" workloads are really count(*) with a WHERE clause and those never touch the source at all.

Store what you display. The common setting. Include the title, the url, the price, the timestamp: whatever a result card renders. Searches serve entirely from the index; a user who clicks through to a full record pays one keyed read.

Store everything. Now the index is a self-contained local mirror and the source is only consulted when you rebuild. This is roughly what Elasticsearch does with _source, except Elasticsearch decided it for you, which is part of why the mirror costs what it does.

Concretely. A shop's product table holds title, description and brand, which arrive with a supplier catalogue and change a few times a year, next to price_cents and stock, which change all day and change inside transactions, because you cannot sell stock you do not have. Index the text, include the brand and leave the two numbers where the transactions are:

CREATE INDEX products_fts ON products
  USING inverted(sku, title en, description en)
  INCLUDE (brand);

Search then runs entirely on local disk, matching and ranking and filtering by brand, while the two values a customer actually acts on are read from Postgres as the page renders. The price on the screen cannot be stale, because we do not have it. When the index falls behind, a product from yesterday's catalogue import is not findable yet. Nobody is ever shown a wrong price.

The reason to include a column is that you do not want the round trip. Everything you leave out is read from the source when a query asks for it, so put the columns your result page renders into the index and the search stays on local disk.

There is also a limit worth knowing about. The connector can only go back to a view that reads one source plainly. Put a join or a GROUP BY in the view body, say SELECT customer_id, string_agg(body, ' ') FROM tickets GROUP BY customer_id and reading real columns is refused outright with materialising real columns from this view-backed inverted index is not yet supported. That index still matches, counts and ranks. It just cannot hand you the rows.

What that buys you​

No second copy of the corpus. You store an index and you choose per column how much more than that you want.

No pipeline. There's no CDC job, no queue, no mapping template drifting out of sync with a schema, no dashboard panel asking whether the sync is behind. The build is a DDL statement.

Adoption without a migration. This is the part platform teams actually care about. Your warehouse stays where it is, keeps its ingest, its retention rules and its access control while gaining a capability from the outside. If it doesn't work out, DROP SERVER puts you back exactly where you started. That's a much easier conversation than "we should move the data."

Results that are relations. The matched set is a table like any other, so you can aggregate it, window it, join it to a local dimension table, park it in a CTE or rank it by a blend of BM25 and vector distance. Over rows that are physically in ClickHouse.

Keep the database you have. Add search on top of it. Skip the copy in between.

How the lookup works​

Four things happen between CREATE INDEX and a row coming back.

The remote engine is touched in three different ways and only one of them repeats. Reading the primary key and streaming the table happen once, when the index is created (dashed). After that the only traffic is the keyed fetch for columns that are neither indexed nor included, one statement per batch of matches.

How a row is identified differs by engine and each default follows from how that engine works. Postgres keys on ctid, the physical row location: universal, no primary key required and the fetch pushes down as a TID scan. ClickHouse keys on the table's MergeTree primary key instead, read from system.columns, because part-and-offset ids do not survive a merge. Either default can be overridden with WITH (key_columns = ...).

The statements differ too, since each planner recognises a different shape: on Postgres the keys travel as a bound array parameter, on ClickHouse as a native columnar block sent alongside the query. Rows come back in whatever order the engine produced them and are put back in place by an ordinal that travelled out with the batch.

What else the connection is good for​

A search index is one use of a foreign server. The connection underneath has a few others.

One index over several engines. An index can be built over a view and a view can be a union of tables from different servers. Fresh documents in Postgres, the archive in ClickHouse, cold data as Parquet in object storage and a single index across all of it. One query, one ranking and no stitching results back together in application code. A union has no single table to key on, so this is the keyless mode described above — postings key on a synthetic row id and you include the columns you want back.

Joins across engines. Facts in ClickHouse, dimensions in Postgres, one statement. Neither side is pre-joined or copied and the search index can sit on either of them.

ETL without the tool. Both connectors write as well as read, so INSERT INTO ... SELECT across servers is one statement: Postgres into ClickHouse or either of them into SereneDB's own storage once you decide the data should be local after all. The extract can be driven by a search query, which turns "everything matching this, into that table" into a one-liner — a document export, a training-set build, a subset migration.

All three pull their data through one process, so the scale they suit is a nightly job, a subset or a dimension table. Hundreds of terabytes need a real pipeline.

What comes next: ODBC and ADBC​

Two wrappers cover two engines. The next two cover most of the rest.

ODBC is the lowest common denominator of database connectivity. If a database has been sold commercially in the last thirty years, it has an ODBC driver. A wrapper shaped like clickhouse_fdw but speaking ODBC points CREATE SERVER at anything with a DSN: MySQL, SQL Server, Oracle, Snowflake, Redshift, Databricks, anything with a DSN.

ADBC is the newer Arrow-native one and it is the better fit for what we do. Result sets arrive as Arrow columnar batches instead of one row at a time. That matters most during an index build, where we stream a whole table exactly once. Postgres, DuckDB, Snowflake, BigQuery and Flight SQL all have ADBC drivers already.

Neither one changes anything above the connector layer. The index, the INCLUDE columnstore, the key-based fetch and the SQL you write all stay as described above. And for sources where no primary key can be discovered from metadata, which becomes a good deal more likely once you are going through a generic driver, the keyless path already exists: postings key on a synthetic row id and the columns you included come back from the local columnstore.

You end up with one CREATE SERVER surface and one index mechanism. The list of engines you can reach becomes the list of engines that ship a driver.

Smaller items on the connector itself: fuller batch accumulation on the lookup path and parameterized IN {keys:Array} over the ClickHouse native protocol.

Everything here is on main, Apache 2.0.

curl https://install.serenedb.com | sh

Point a CREATE SERVER at a database you already run, build an index on a table that isn't ours and tell us what breaks.

Six years ago, when we first shipped fuzzy search (yes, without agents!), we wrote up the two algorithms that sit underneath it. The search engine has the same name but a new home since then. IResearch is now the search core of SereneDB and you reach it through SQL instead of a bespoke query language but the theory behind the implementation has not aged a day. This is that article, rewritten for where the code actually lives today.

"Fuzzy search" is an umbrella term for a family of approximate-matching algorithms. Each one defines some similarity measure between a query term and the terms in a dictionary, so the engine can decide which results are close enough to show and in what order. In this post I'll walk through the two that matter most, because they are genuinely different tools for different jobs:

• Approximate matching based on Levenshtein distance
• Approximate matching based on n-gram similarity

I'll go deep on each, flag the problems you hit when implementing them at scale and show how both are exposed in SereneDB today.

Why fuzzy search at all?​

We deal with unstructured, imperfect text everywhere. Web search taught a whole generation that being inexact is normal: you fat-finger a query on a phone and expect the typo to be fixed for you. But it goes well beyond autocorrect:

• Linguistics. Identifying cognates across dictionaries is central to historical linguistics and cognates are, by definition, not identical.
• Bioinformatics. A DNA sequence is an absurdly long string over a four-letter alphabet (ACGT); quantifying variation between sequences is approximate string matching.
• Records and search. Deduplicating people, products or addresses means matching "Jon Smith" to "John Smith" and "priorty_queue" to "priority_queue".

All of these need a way to say how close two strings are. Let's start with the most famous answer.

Approximate matching based on Levenshtein distance​

The Levenshtein distance between two words is the minimum number of insertions, deletions or substitutions needed to turn one into the other.

For example, the distance between foo and bar is 3, because every letter has to be substituted:

foo -> boo -> bao -> bar

Formally, the distance between strings $a$ and $b$ (of lengths $|a|$ and $|b|$) is $\operatorname{lev}_{a,b}(|a|, |b|)$, where:

Treat Levenshtein distance as our relevance measure and the goal becomes: for a given input, find the closest terms in the dictionary.

The classic Wagner–Fischer algorithm computes this with dynamic programming in $O(|a| \cdot |b|)$ time, with memory linear in the shorter string if you keep one row at a time. That's fine for comparing two words and hopeless at "web scale", where a real dictionary holds hundreds of thousands of terms and you'd have to run it against every single one.

The automaton trick​

In 2002, Klaus U. Schulz and Stoyan Mihov published a beautiful result: for any fixed distance $n$ and input word $W$ of length $N$, you can build a deterministic automaton $A(W)$ that accepts every string within Levenshtein distance $n$ of $W$ and you can build it in time and space linear in $N$.

Once you have $A(W)$, you intersect it with the term dictionary. If the dictionary is a trie (or an FST), the automaton walks the tree and prunes entire subtrees the moment they can no longer lead to an accepting state. Say the dictionary holds avocado, avalon, avalanche and cargo:

Searching for terms within distance 1 of kargo, the automaton never descends into the av- subtree: aligning the query's k with the dictionary's a already costs one edit and the next character (a against v) forces a second, which exceeds the limit of 1. The whole left branch is pruned and only cargo survives. We touch a handful of nodes instead of scoring the entire dictionary.

So the real task is:

Given an input word $W$ of length $N$ and a maximum edit distance $n$, build a deterministic finite automaton (DFA) that accepts a word $V$ iff $\operatorname{lev}(W, V) \le n$.

Building it up from an NFA​

It's easier to first draw a non-deterministic finite automaton (NFA) and worry about determinizing later. Here is the NFA for foobar at distance 1:

Read it as a grid. The bottom lane (e=0) is "no edits spent yet"; the top lane (e=1) is "one edit spent". Each state is an (index, edits) pair. Four families of transitions leave a state $I^{J}$ (index $I$, $J$ edits):

• Match: $I^{J} \to (I{+}1)^{J}$, consuming the correct next character of $W$. These are the horizontal edges.
• Insertion: $I^{J} \to I^{J+1}$. An extra character in the candidate, so spend one edit and stay at the same position.
• Substitution: $I^{J} \to (I{+}1)^{J+1}$. Wrong character, so spend one edit and advance.
• Deletion: $I^{J} \to (I{+}2)^{J+1}$. A character of $W$ is missing from the candidate, so skip it. (Deleting $K$ consecutive characters reaches $(I{+}K{+}1)^{J+K}$.)

The moment you spend an edit you move up a lane and at distance 1 the top lane only has match edges left, because you're out of budget. That's why we say the flow "transfers to the upper lane" as soon as an edit happens.

Raising the budget to 2 just adds a third lane and the deletion edges that skip two characters at once (an $I^{J} \to (I{+}3)^{J+2}$ family). Each state gains $O(n)$ outgoing edges. This is the pattern we now need to bound.

Making it linear​

The naive powerset construction for the DFA gives $O(2^{(n+1)N})$ states. It's clearly far too loose given how regular the NFA is. Schulz and Mihov tightened it in three steps:

1. Locality. From position $i$ you can never reach past $i+n$ by inserting, nor before $i-n$ by deleting. So at most $2n+1$ NFA states are ever "live" at a given position and the powerset over them has $2^{2n+1}$ members. That alone drops us to $O(2^{2n+1} N)$ states.
2. Subsumption. State $I^{J}$ subsumes any $(I{\pm}K)^{J+K}$ with $K \le n - J$: the other state sits up to $K$ positions away and paid $K$ extra edits to get there, so everything it can still accept the cheaper state accepts too. Keeping it around is pointless. This removes the exponential factor entirely: $O(n^2 N)$ states.
3. Parametrization. The transitions depend only on the distribution of the current character relative to position $I$, not on $I$ itself. That kills the dependency on $N$, the last thing standing between us and linear time.

The key object in step 3 is the characteristic vector. For a character $c$, $\chi(c, W, I)$ is a bit set of length $\min(2n+1,\ |W|-I)$ whose bit $k$, counting from 0, is 1 iff $W_{I+k} = c$. Index $I$ means $I$ characters consumed, so $W_{I}$ is the character up next and bit 0 tells you whether the match transition is available at all. It answers "where, in the next few characters of $W$, does $c$ appear?" For foo:

The last one is a single bit because only one character of foo is left to look at.

Because only $2n+1$ states matter at any position, we can enumerate all $2^{2n+1}$ possible character distributions and, for each, list which states become reachable. Doing that for distance 1 turns up just five distinct reachable state sets, the parametric states:

Now the DFA transition function $\Delta$ is a small lookup keyed by (parametric state, characteristic vector). For distance 1 it fits in one table:

Schulz and Mihov generalized $\Delta$ for arbitrary $n$. Build it once and you can then instantiate the DFA $A(W)$ for any word $W$ in a single linear pass over its characters. We went from $O(n^2 N)$ down to $O(N)$ and that is what makes Levenshtein search practical. In IResearch $\Delta$ is ParametricDescription and DefaultPDP hands out one lazily built instance per (distance, transpositions) pair, nine slots in total. The per-term DFA comes out of MakeLevenshteinAutomaton, wrapped by the ByEditDistance filter.

They also showed that a tiny addition to $\Delta$ buys you Damerau–Levenshtein distance, which treats a transposition of adjacent characters as a single edit. foobar → foobra is distance 2 under plain Levenshtein but distance 1 under Damerau–Levenshtein, much closer to how humans actually mistype.

How a fuzzy query runs​

Putting it together, a ts_levenshtein query in SereneDB flows like this:

The dictionary walk hands back more than a list of terms. Each accepted term carries the distance it was accepted at and the filter turns that into a per-term boost of $1 - d / \min(|V|, |W|)$. An exact hit therefore scores above a term accepted at the maximum distance, so the accepted terms rank against each other instead of all arriving with the same score.

Caveats​

$\Delta$ stores one transition per (parametric state, characteristic vector) pair, so its size is the number of parametric states times $2^{2n+1}$. The state count is what explodes with $n$:

• $n=1$: $5 \cdot 2^{3} = 40$ transitions
• $n=2$: $30 \cdot 2^{5} = 960$
• $n=3$: $196 \cdot 2^{7} = 25{,}088$
• $n=4$: $1{,}353 \cdot 2^{9} = 692{,}736$

That growth makes very large distances impractical, so SereneDB caps the edit distance at 4 for Levenshtein and 3 for Damerau–Levenshtein. Beyond that "fuzzy" stops meaning anything useful anyway.

There's also a lovely trick for squeezing one more unit of distance out of a smaller automaton. The edit distance is invariant under reversal: $\operatorname{lev}(W, V) = \operatorname{lev}(W', V')$ for the reversed strings. So by keeping two term dictionaries, one forward and one reversed (a so-called FB-trie), you can answer distance-$(n{+}1)$ queries using two distance-$n$ automata. Given how fast the DFA grows with $n$, that's a very good trade.

Trying it in SereneDB​

All of the above is behind a single SQL function, ts_levenshtein. First, a text dictionary and an inverted index over some product names:

CREATE TEXT SEARCH DICTIONARY fuzzy_dict (
    template = 'text',
    locale = 'en_US.UTF-8',
    case = 'lower',
    stemming = false,
    accent = false
);

CREATE TABLE products (id INTEGER PRIMARY KEY, name VARCHAR);

CREATE INDEX idx_products ON products
    USING inverted (id, name fuzzy_dict);

INSERT INTO products VALUES
    (1, 'cat'), (2, 'bat'), (3, 'car'),
    (4, 'dog'), (5, 'cats'), (6, 'act');

VACUUM (REFRESH_TABLE) products;

Fuzzy matching is just the @@ operator against a ts_levenshtein acceptor:

-- Distance 1: everything one edit from 'cat'
SELECT id, name FROM idx_products
WHERE name @@ ts_levenshtein('cat', 1)
ORDER BY id;
--  1 cat | 2 bat | 3 car | 5 cats | 6 act

If you drop the distance argument you get auto mode then. It picks the distance from the query length: 0 for two characters or fewer, 1 for three to five, 2 from six up.

-- No distance: 'cat' is 3 characters, so distance 1
SELECT id, name FROM idx_products
WHERE name @@ ts_levenshtein('cat')
ORDER BY id;
--  1 cat | 2 bat | 3 car | 5 cats | 6 act

This is the form you want behind a search box, where the query grows one keystroke at a time. A fixed distance of 2 is fine for catalogue and useless for ct, since at two characters almost every short token in the dictionary is within two edits. Auto mode gives ct distance 0 and catalogue distance 2 without you branching on length() in SQL.

act matches because a transposition counts as one edit. Transpositions are on by default (Damerau–Levenshtein). Turn them off and act drops out:

-- Strict Levenshtein: 'act' is now distance 2 from 'cat'
SELECT id, name FROM idx_products
WHERE name @@ ts_levenshtein('cat', 1, false)
ORDER BY id;
--  1 cat | 2 bat | 3 car | 5 cats

You can also anchor a literal prefix and only fuzz the tail. That's cheap, because the prefix walks the trie directly and the automaton only kicks in afterward:

-- Must start with 'ca', fuzzy-match the rest within distance 1
SELECT id, name FROM idx_products
WHERE name @@ ts_levenshtein('t', 1, true, 'ca')
ORDER BY id;
--  1 cat | 3 car | 5 cats

Because it's an ordinary acceptor, it composes with the rest of SQL.

SELECT id, name FROM idx_products
WHERE name @@ ts_levenshtein('cat', 1) AND id < 4
ORDER BY id;
--  1 cat | 2 bat | 3 car

Spell correction​

Being ordinary SQL also allows you to get the matched terms themselves. Point ts_levenshtein at a query log and read the accepted dictionary entries back with the ts_dict aggregates:

SELECT unnest(ts_dict_agg(term))   AS suggestion,
       unnest(ts_dict_score(term)) AS similarity,
       unnest(ts_dict_count(term)) AS searches
FROM query_log_idx
WHERE term @@ ts_levenshtein('jaket', 2)
ORDER BY similarity DESC, searches DESC;
--  jacket | 0.8 | 5
--  basket | 0.6 | 1
--  racket | 0.6 | 1

ts_dict_agg returns the terms the automaton accepted, ts_dict_score is the $1 - d / \min(|V|, |W|)$ similarity from the dictionary walk and ts_dict_count is the indexed frequency. Sorting by similarity first and frequency second is what turns three candidates at distance 2 into one correction: jacket was searched 5 times, the other two once each. Add LIMIT 1 and you have a "did you mean".

Why another kind of fuzziness?​

Raw edit distance isn't always the right lens. Two problems show up quickly.

First, length bias. A distance of 2 means something very different for a 4-letter word than for a 20-letter one. The obvious fix is to divide the distance by the length and flip it into a similarity:

$\max$ is what keeps the result inside $[0,1]$. Divide by $\min$ and the ratio can exceed 1 (a against xyz gives $3/1$), which is useless as a score. Even with $\max$ you don't get much: the maximum edit distance we can afford is small, so every long string that clears the cap lands a couple of percent below similarity 1 and the score has nothing left to discriminate with.

Second, phrases. Allowing one edit per word lets quck brwn fx match quick brown fox. But it will never match quick-witted brown fox, because per-word edit distance has no notion of extra or missing words. For that we want a measure that tolerates extra and missing material in the indexed value.

Approximate matching based on n-gram similarity​

A different way to compare two strings is by their longest common subsequence (LCS) of characters: the longer the LCS, the more similar. On its own, character LCS is too context-free: connection and fonetica share a 5-character subsequence (oneti) despite meaning nothing alike.

Grzegorz Kondrak's fix is to run the LCS over n-grams instead of single characters, so each unit carries a little local context. Compare the same pair as 3-grams:

connection -> con onn nne nec ect cti tio ion
fonetica   -> fon one net eti tic ica

Now they share zero trigrams and the spurious similarity is gone.

Formally, let $X = \langle x_1 \dots x_k\rangle$ and $Y = \langle y_1 \dots y_l\rangle$ be sequences over a finite alphabet. Write $\Gamma_{i,j}$ for a pair of prefixes, $\Gamma^{*}_{i,j}$ for a pair of suffixes and $\Gamma^{n}_{i,j}$ for a pair of n-grams starting just after positions $i$ and $j$. The base case, comparing two single n-grams, is binary:

The similarity of the full sequences is then an LCS-style recurrence over n-grams:

Normalize by the longer string to land in $[0, 1]$ and shed the length bias:

The practical beauty of this approach is that it needs no per-query automaton. You split each indexed term into n-grams, store those n-grams as terms in the dictionary and record their positions within each document. At query time you split the input the same way, look up the posting list for each n-gram and for every matched document use the recorded positions to compute the n-gram LCS and finally $s_N(X, Y)$.

That two-step shape, a cheap posting-list lookup to gather candidates followed by an exact positional check, is exactly the two-phase execution model IResearch uses across phrase, geo and nested queries. Phase 1 requires at least k of the query's n-grams to be present, which is a MinMatchDisjunction wrapped as NGramApprox in ngram_similarity_query.cpp. Phase 2 decodes positions to confirm they actually line up and it only runs for documents phase 1 already accepted.

That k is where the implementation differs from Kondrak's $s_N$. MinMatchCount derives it as $k = \lceil |X| \cdot t \rceil$ for threshold $t$ and query n-gram count $|X|$, so the score is normalized by the query alone and $|Y|$ never enters the picture. Extra material in the indexed value is free. ts_ngram('hello', 0.7) matches a title helloworld because all four query bigrams line up in order, while $s_N$ scores that pair $4/9$ and rejects it. That asymmetry is what lets a short query find a long field. It also gets you the phrase case from earlier: over a character-bigram dictionary ts_ngram('brwn fox') matches quick-witted brown fox at the default threshold, the leading extra word costing nothing. What you still pay for is typos, since each one takes out the bigrams that straddle it. quck brwn fx keeps 7 of its 11 bigrams against that title, so it needs the threshold down at 0.6.

A sharper score​

Kondrak also proposed a refinement: instead of the strictly-binary base case, count how many individual characters line up inside a matched n-gram window:

Depending on the corpus this bought Kondrak 10–20% better accuracy. The catch is that it's harder to evaluate. Here the two algorithms in this post meet: split the query into n-gram tokens, build a Levenshtein automaton for each, union them into one big automaton and use that to pull all the near-miss n-grams out of the dictionary in one sweep. For each matched n-gram you then measure the per-character similarity. Fuzzy matching all the way down.

Trying it in SereneDB​

N-gram similarity needs an ngram dictionary that records both frequency and positions:

CREATE TEXT SEARCH DICTIONARY bigram_dict (
    template = 'ngram',
    mingram = 2,
    maxgram = 2,
    frequency = true,
    position = true
);

CREATE TABLE articles (id INTEGER PRIMARY KEY, title VARCHAR);

CREATE INDEX idx_articles ON articles
    USING inverted (id, title bigram_dict);

INSERT INTO articles VALUES
    (1, 'hello'), (2, 'help'), (3, 'world'), (4, 'held'), (5, 'hero');

VACUUM (REFRESH_TABLE) articles;

Then ts_ngram takes a similarity threshold in $[0, 1]$ (default 0.7), measured against the number of n-grams in the query:

-- Strict: only near-identical titles
SELECT id, title FROM idx_articles
WHERE title @@ ts_ngram('hello')
ORDER BY id;
--  1 hello

-- Loosen the threshold and neighbours appear
SELECT id, title FROM idx_articles
WHERE title @@ ts_ngram('hello', 0.3)
ORDER BY id;
--  1 hello | 2 help | 4 held

Summary​

Neither algorithm wins outright. Levenshtein automata give you a precise, bounded notion of "within k typos", built in linear time and pruned against the term dictionary, which is perfect for autocorrect-style matching and short terms. N-gram similarity gives you a normalized score that survives extra and missing characters and rides the ordinary inverted index with no per-query construction, which is better for longer strings and partial overlap. Most real systems reach for both and sometimes combine them.

References​

• Wagner–Fischer algorithm
• Schulz & Mihov, Fast string correction with Levenshtein automata, IJDAR 5(1):67–85, 2002
• Trie | NFA | DFA
• Damerau–Levenshtein distance
• N-gram | Longest common subsequence
• Kondrak, N-gram similarity and distance, SPIRE 2005, LNCS 3772:115–126 (doi)
• IResearch source: levenshtein_utils | levenshtein_default_pdp | levenshtein_filter | ngram_similarity_filter | ngram_similarity_query | disjunction

IResearch is open source (Apache 2.0) and available as part of SereneDB. If you find this work interesting, starring us on GitHub goes a long way for an early-stage project.

We needed search on our docs. We make a search database. You can see where this went.

docs.serenedb.com and this blog now run on SereneDB-backed search. You can try it right here: press Cmd+K on macOS or Ctrl+K elsewhere to open the search. Then we packaged the whole path from "these are my docs" to "there is a search box on the site" into Serene Docs Search.

The short version is: point it at a Git repository, a local folder, a live HTML site or an S3-compatible bucket; choose full-text or hybrid search; download the generated config and Compose file; add one React component or script tag to the site. It keeps the index in sync after that. Ask AI can answer questions with citations. An optional MCP server lets Codex, Claude and other agents search and read the same index.

For straightforward documentation and website search, it is a small self-hosted alternative to Algolia DocSearch. It is not trying to reproduce every feature of a search SaaS. It covers the common case, keeps the stack and data under your control and the application packages are licensed under Apache 2.0. Full-text mode does not need an embeddings service or a model API, so the only bill is the infrastructure you choose to run it on.

From source to search box​

Documentation does not have one standard home. It may be Markdown in the main repository, generated HTML on a website, a directory mounted into a build server or exported files in object storage. Serene Docs Search treats all four as first-class sources:

• Git: follow a branch, pin a commit and limit indexing to selected files or subdirectories.
• Local folder: mount any directory read-only into the backend.
• Website: crawl from a URL, follow a sitemap and choose a maximum link depth.
• S3: index a bucket or prefix, including S3-compatible services such as R2 and MinIO through a custom endpoint.

The indexer understands Markdown, MDX, HTML, reStructuredText, plain text, Jupyter notebooks and PDFs. Markdown can stay as one result per file or split at headings so a result links directly to the relevant section. HTML extraction can be scoped to article, main or your own CSS selectors, with navigation and other boilerplate excluded before indexing.

Source paths and public URLs are often different, so URL mapping is part of the configuration rather than application glue. You can strip source prefixes and file extensions, map README and index files to directory URLs or route different paths to different sites. One repository can serve a docs domain and a blog domain without giving either one broken result links.

Git / folder / website / S3  ── pull, parse, sync ──┐
                                                    ▼
browser widget  ── /v1/search, /v1/ask ──>  search backend  ──>  SereneDB
                                                    │
AI agents      ── MCP tools ────────────────────────┘

The setup wizard asks for the source, parsing rules, search mode and sync policy, then generates serene-search.config.json and docker-compose.yml using the same code as the backend packages. The standard stack is two containers: SereneDB and the search backend. Ollama and the MCP service appear only when you enable them.

You can configure a real stack from this page now. For a walkthrough of every option and deployment step, follow the step-by-step guide in Apps and clients. The wizard saves an unfinished draft in this browser, but it does not change the search configuration of this site.

Search first; AI stays optional​

The default job of a docs search box is to find a page while somebody is still typing. Serene Docs Search starts there.

Full-text mode uses BM25 with extra weight on titles. All complete query terms must match, while the last term works as a prefix for search-as-you-type. The analyzer lowercases and folds accents, removes stop words and stems terms, so run can find running. You can add Solr-format synonyms such as db, database or k8s => kubernetes.

Relevance gets a few deliberately practical passes on top:

• exact and prefix title matches stay above looser matches;
• one long page cannot occupy the entire result list with its headings;
• configured pages can be pinned for matching queries;
• if a strict query returns nothing, a typo-tolerant pass tries again and can show a Did you mean correction;
• partial matches are used as a last fallback instead of returning an empty modal;
• every lexical hit includes a snippet with the matching terms highlighted.

If keyword search is enough, stop there. There are no vectors to generate and no model endpoint to operate.

Hybrid mode adds vector similarity over the same sections and merges the lexical and semantic rankings with reciprocal rank fusion. The embeddings provider can be OpenAI, Ollama or another OpenAI-compatible endpoint. The fusion weight, candidate window and distance threshold are tunable per corpus.

Search results can also be grouped into sections such as Docs, Blog and API reference. The group matching the page a visitor is currently reading moves first, while relevance order stays unchanged inside every group. Empty groups disappear. This is useful when one search backend serves several related sites without making users choose a scope before they search.

When a user selects a result, the widget navigates to the heading anchor, flashes the target heading and highlights the query terms on the destination page. Query and click counters are recorded too, with an admin-only report for top queries, zero-result queries and most-clicked pages.

Ask the same index a question​

Ask AI is a separate switch, not a requirement for hybrid search. When it is enabled, the model searches and reads sections through tools over the same index, streams the answer into the modal and cites the pages it used. Multi-turn history stays in the conversation. Providers without tool calling fall back to a single retrieval-and-answer pass.

The answer model and embeddings model are configured independently. You can use an OpenAI-compatible API for one, local Ollama for the other, use Ollama for both, or leave Ask AI off completely. API keys stay in backend environment variables; they are never shipped with the browser widget.

The docs can be tools for agents too​

A useful documentation index should not only serve humans. The optional @serenedb/docs-search-mcp server exposes three tools:

• search_docs finds relevant sections;
• read_section returns the full text behind one result;
• docs_health reports whether the corpus is indexed and current.

It uses the same public HTTP API as the widget, so there is no second ingestion pipeline or agent-specific copy of the docs. The modal can show a ready-to-copy setup command for Codex or Claude. The configurator can also add the MCP container to the generated Compose stack. Ask AI does not depend on MCP; the two features remain independent opt-ins.

Keep it current without rebuilding everything​

The backend owns synchronization as well as queries. For Git it can cheaply watch the branch for a new commit. Any source can be polled on an interval. CI can instead call the reindex webhook after a deployment.

Snapshots hash every parsed section. Unchanged sections are skipped, changed ones are replaced and sections deleted from the source are pruned from the index. The setup flow shows fetch, parse, embed and index progress during the initial build, and the health endpoint reports document counts and the last successful sync.

Administrative routes for configuration and manual sync are protected by a token that the Compose generator creates for you. Search, health and Ask AI remain public, so there is no admin secret in the website bundle.

Put the UI where you need it​

For React, the complete integration is a component and a stylesheet:

npm install @serenedb/docs-search-react@latest

import { SereneDocsSearch } from "@serenedb/docs-search-react";
import "@serenedb/docs-search-react/styles.css";

<SereneDocsSearch backendUrl="https://search.example.com" />;

The component includes the trigger, modal, keyboard navigation, Cmd/Ctrl+K hotkey, automatic light/dark theme, recent queries, connection status, Search, Ask AI and MCP tabs. It can hand navigation to an SPA router, use a custom trigger, run as a controlled modal or render into a custom portal.

If you already have a search interface, useSereneDocsSearch() exposes the same connection, query, grouping, navigation and streamed-answer state without our UI. If you do not use React, the embed package bundles the widget behind a stylesheet and one script tag, with both explicit and zero-JavaScript initialization.

Underneath those integrations is a small HTTP API for health, search, streamed answers, reading complete sections, synchronization and redacted configuration. The widget is optional; the index is usable from any client that can make an HTTP request.

What “free alternative” means​

Serene Docs Search is deliberately for the simple case: a site or docs corpus, a search modal, a small backend and one database. If you need a global managed search platform with a large operations team behind it, use one. If you need good search over your own docs and would rather keep the source, index, relevance configuration and deployment in your own repository, this is the case we built for.

The software is open source and licensed under Apache 2.0. There is no per-search fee and full-text mode has no model cost. Self-hosting still means providing the machine, storage and operations. Optional model APIs charge whatever their providers charge. The trade is explicit: less SaaS surface area, more ownership.

That is also why the configurator generates plain files instead of creating a project in somebody else's control plane. You can inspect them, commit them, change them and deploy them anywhere Docker Compose runs.

Try the search on this site, open the complete configuration guide, or generate your own stack with Configure now above.

If Serene Docs Search is useful to you, star SereneDB on GitHub and tell us what you would like it to do next.

Welcome back to The State of Serene, our monthly note on what shipped and where SereneDB is heading.

July was a big month. Plenty of new features: term-dictionary facets and autocomplete, partial indexes, an online CREATE INDEX that doesn't block writes, real Postgres roles and grants, CREATE SERVER with a ClickHouse connector behind it and a docs search box that runs on SereneDB. Plenty of performance work too: vector search rebuilt on IVF with RaBitQ queries up to 49x faster, filters pushed down into the index scan and FSST+ compression taking 23.8% off URL columns on disk. We also published SearchBench and came out 16.1x ahead of ArangoDB across 92 queries.

All of it is in v26.07.5, so everything below is something you can pull and run today.

What happened in July​

Faceted search and autocomplete without touching a document​

Every search UI needs the same three things next to the results list: a facet sidebar with counts, a type-ahead and a distinct-value list. On a normal database all three mean scanning the matched documents and aggregating them, which is exactly the work you already paid for once when you built the index.

The inverted index already knows every value of every field and how often each one occurs. That's what a term dictionary is. SereneDB now makes it readable from SQL. The ts_dict_* aggregates enumerate a field's dictionary straight off disk, with no document scan and no postings. The list forms come out positionally aligned, so unnest zips them into rows:

SELECT unnest(ts_dict_agg(body))   AS term,
       unnest(ts_dict_count(body)) AS docs,
       unnest(ts_dict_freq(body))  AS freq
FROM   docs_idx
ORDER  BY term;

Most of the time you don't have to spell any of that. On keyword columns the optimizer rewrites ordinary SQL onto the same path: count(DISTINCT col), min, max, array_agg(DISTINCT col) and the plain SELECT col, count(*) ... GROUP BY col facet shape. EXPLAIN prints TsDict: on the scan when it fires and the document lookup disappears from the plan.

The sidebar case gets its own trick. A real facet panel counts four or five dimensions at once and nobody wants five round trips for that, so GROUPING SETS of single-column sets collapse into one dictionary pass:

SELECT category, brand, price_band, count(*)
FROM   products_idx
WHERE  description @@ 'wireless'
GROUP  BY GROUPING SETS ((category), (brand), (price_band));

The plan tells you it worked. No document lookup, no scan node under the aggregate, just the term dictionary being read three ways at once:

╭─ IRESEARCH_SCAN ────────────────────╮
│ Index: products_idx                 │
│ Index Filter:                       │
│ ╭─ Term ─────────────────────╮      │
│ │ Field: description(string) │      │
│ │ Value: wireless            │      │
│ ╰────────────────────────────╯      │
│ TsDict: category, brand, price_band │
│ ~7 rows                             │
╰─────────────────────────────────────╯

That TsDict: line is the tell. Projections trimmed for width, the rest is what EXPLAIN prints.

Facets survive a WHERE on an INCLUDE'd column now too. All of it works the same on view-backed indexes.

Docs: term dictionary and TSQUERY, plus the faceted search, autocomplete, spell correction and tag cloud recipes.

Vector search, rebuilt on IVF​

SereneDB replaced HNSW with a native IVF index built on its own columnstore. Three reasons.

It scales with disk. An HNSW graph wants to live in memory. Every hop is a random access into the graph, so the moment it outgrows RAM your latency falls off a cliff and index size turns into a hardware budget. IVF keeps the coarse centroids resident and leaves everything else on disk, read lazily per probed cluster. The dataset gets to be much bigger than the box.

It's the same machinery as the rest of the index. A cluster is a term. Its members are that term's postings list and the quantized codes ride along in the postings payload stream. IVF isn't a second index bolted onto the side, it's the inverted index doing its usual job with different keys, so it inherits the on-disk format, the segment lifecycle, compaction and recovery instead of reimplementing all four.

Which is why it combines. A vector query is a disjunction over cluster iterators, so it composes with full-text filters, boolean predicates and column filters in the same scan. No candidate-generation step feeding a filtering step in another system. That's what makes filtered kNN and hybrid search one query instead of an integration.

Declare it inline on the column:

CREATE INDEX idx ON t
  USING inverted(pk, emb ivf (metric = 'l2', quant = 'sq8'));

metric is l2, l1, ip or cosine, matched by the <->, <#> and <=> operators. quant is sq8, sq4, pq, rabitq or none, so you can trade precision for a footprint that fits in memory. You don't size the cluster count by hand any more. A multi-level centroid tree gets built from the row count and sdb_ivf_posting_size moves the target leaf size if you want to push on it.

Recall then costs you two session settings at query time rather than a rebuild. sdb_nprobe decides how many clusters a query scans and sdb_rerank_factor sizes the pool that gets re-scored with exact distances, so the same index serves a cheap approximate query and an expensive high-recall one. kNN is still plain SQL. An ORDER BY <distance> LIMIT k gets you the nearest k and a WHERE <distance> < threshold gets you everything inside a radius.

Builds come out around 60x faster than the HNSW index they replace.

RaBitQ was the slowest quantizer. Its 1-bit default was the worst case: one scalar distance call per vector per probed cluster. That's a single PQ4 fast-scan per cluster now, at every bit width. Measured on dbpedia, 100K vectors at 1536 dimensions, l2, k=10:

Recall came out the same or better, build time and index size unchanged.

RaBitQ and PQ both rebuilt their entire scoring pipeline per probed cluster: roughly 10 to 20 µs of setup against 0.07 µs per vector of scanning. The code-dependent half is a lookup table built once per query now, so latency stays nearly flat as nprobe grows. 20% faster at nprobe=128, 24% at 512.

Filtered kNN returns the right rows now. An ORDER BY emb <-> $1 LIMIT k with a WHERE on an INCLUDE'd column pushes the filter into the scan instead of running it above.

Those are SereneDB's own before-and-after numbers, which only tell you it got faster than it was. For where SereneDB lands against somebody else, there are preliminary recall-versus-QPS results against Qdrant at serenedb.github.io/vector-search-benchmark. Treat them as early. Proper vector benchmarks are part of benchmark season and they're coming.

One upgrade note, because it matters: the payload layout changed twice in July. Vector indexes built before v26.07.5 need a rebuild.

Docs: vector search.

CREATE INDEX stopped blocking your writes​

CREATE INDEX ... USING inverted(...) no longer blocks writes. Inserts, updates and deletes run against the table for the whole build and the index that gets published matches the table exactly. The backfill is parallel, so it gets faster with more cores. Deletes that race the build are handled, so nothing removed mid-build survives in the finished index.

Partial indexes handle the case where only some rows are worth indexing:

CREATE INDEX recent_errors ON logs USING inverted(message log_dict)
  WHERE level = 'error';

PostgreSQL semantics all the way down, including NULL counting as non-matching. Rows enter and leave the index as updates move them across the predicate boundary, so membership stays correct without a reindex.

Index tuning is real DDL now:

CREATE INDEX opt_tuned ON opt_t USING inverted(label)
  WITH (segment_memory_max = 67108864, segment_docs_max = 5000,
        compaction_max_segments = 4, compaction_floor_segment_bytes = 4194304);

ALTER INDEX opt_tuned SET (segment_memory_max = 33554432);
ALTER INDEX opt_tuned RESET (segment_memory_max);

ALTER INDEX SET applies live to the running storage, writer limits and task cadences both. pg_class.reloptions lists the persisted set and pg_index.indisvalid / indisready are false while a build is in flight, so anything already speaking Postgres can see what's going on.

Docs: inverted CREATE INDEX.

Filters push all the way into the scan​

"Top 10 relevant errors from the frontend service in the last hour" is two problems glued together: a search and a filter. If the filter runs above the scan you score every match and then throw most of it away, which gets worse the more selective your filter is. That's backwards.

Index scans accept pushed table filters now. Covered columns filter inside the scan at codec level, with a row-group zonemap short-circuit and one decode pass that doubles as materialization, so a filtered top-k only ever collects rows that pass. New BitpackingFilter and AlpFilter skip whole codec groups whose header bounds already refute the predicate, without decoding them at all.

There's one new knob and it's a good one: a static score > c bound is consumed at pushdown. The top-k collector starts at that floor, streaming WAND seeds its threshold from it, the filter vanishes from the plan and EXPLAIN prints Min Score.

The numbers, all measured on our side:

• dense lookups against a parquet-backed view: from 40-66x the cost of a native scan down to 1.6-3.1x
• ORDER BY x ASC LIMIT 5 on 1M rows: 0.219 ms to 0.128 ms, 42% off
• a deliberately adversarial DESC TopN over insert-ordered data: 12 ms to 0.6 ms, 8,480 rows scanned instead of 250,000
• impossible filters fold to an empty result at plan time and never touch storage

None of this is search-specific. Plain analytical scans over columnar storage get the same treatment.

Postgres roles and grants, for real​

Up to now SereneDB was effectively single-user. Fine on a laptop, a non-starter the moment two people or a BI tool share an instance. RBAC is the thing that lets you put it in front of a team. It's PostgreSQL's actual model rather than a lookalike, so GRANT, REVOKE, role membership and column-level privileges behave the way your existing scripts already expect.

CREATE ROLE analyst LOGIN PASSWORD 'secret' VALID UNTIL '2027-01-01 00:00:00+00';
GRANT SELECT (id, title, created_at) ON docs TO analyst;
GRANT analyst TO junior_analyst;

Roles, nested membership through GRANT role TO role, column grants, VALID UNTIL expiry, SET ROLE mid-session and SCRAM over the wire.

The defaults are the part worth reading. The whole policy lives in HBA now instead of a pile of per-connection special cases:

• psql -h 127.0.0.1 -U postgres works with no password on a fresh install and over the unix socket too. First connect stays trivial.
• Superuser over the network requires a password. It used to be silently trusted on 0.0.0.0, which is the kind of default that ends up in an incident report.
• Every non-superuser needs a password, local or remote, same as Postgres.
• SERENEDB_INITIAL_PASSWORD=secret on first boot seeds the superuser password, so a container comes up remote-reachable and authenticated in one step.
• Three superuser loopback trust lines get force-prepended to every ruleset, so a broken HBA config can't lock the admin out of the box.

The pre-RBAC --auth_password, --auth_method and --auth_user flags are gone. --auth_password used to shadow RBAC entirely by pinning the server to one hard-coded user. Per-role catalog passwords are the only auth path now.

There's a whole security chapter now, where every example on the page is a query we actually run: roles, privileges, role membership and client authentication. Statement pages too: CREATE ROLE, GRANT and REVOKE.

Index ClickHouse and Postgres with CREATE SERVER​

The data you want to search is usually already sitting in Postgres or ClickHouse and nobody's excited about maintaining a second copy of it.

CREATE SERVER and CREATE USER MAPPING make a remote system a catalog object that survives restart, so it's infrastructure rather than an ATTACH you have to remember to redo every session. There's a new ClickHouse connector behind it too, speaking the native TCP protocol with columnar blocks and LZ4 or ZSTD, with projection, filter and TopN pushdown, a shared connection pool and read plus write support.

The part that makes this more than plumbing: you can build an inverted index over an attached ClickHouse or Postgres table. Search runs here, matched rows get re-fetched from the remote by its own primary key. Full-text and vector search over a ClickHouse table without moving the ClickHouse table.

Credentials never enter SQL text and pg_foreign_server redacts secret values, which is a deliberate divergence from Postgres.

Docs: CREATE SERVER and external data.

String compression: FSST+ and bitpacked RLE​

On log and URL data the string columns are the storage bill. FSST encodes each dictionary entry whole, so the redundancy between neighbouring entries goes unused: a symbol table is global while shared prefixes are local. FSST+ cleaves each entry into a shared prefix plus a suffix and stores every distinct prefix once, which is exactly the shape URLs, paths and stack traces have. On ClickBench hits_10pct:

Reads came out at parity, within 2%, because the prefixes get FSST-decoded once when the segment initializes. Writes cost about 1.4x.

RLE now bitpacks both its streams, run values and run counts, which is 1.3 to 2.4x smaller on run-heavy data, 19 to 34% cheaper on the write path and 12 to 23% faster to read at short run lengths. It also declines outright when every run is a singleton instead of pretending to help.

Deploy it on Kubernetes, build it on macOS​

There's a Helm chart now, published with every release, so getting SereneDB onto a cluster is one command:

helm install mydb \
  https://github.com/serenedb/serenedb/releases/download/helm-chart-v0.0.5/serenedb-0.0.5.tgz

That gives you a single-node StatefulSet on a persistent volume, client and headless Services, generated superuser credentials and a NetworkPolicy. Upgrades recreate the pod on the same PVC, so serened shuts down gracefully and the new version opens the same datadir. helm test mydb runs a SELECT 1 against the service to confirm it came up.

There's deliberately no replicaCount. SereneDB is a single-machine database and the chart gives you exactly one database pod rather than pretending otherwise.

macOS is a supported build target now, which came out of someone asking whether it builds on Apple silicon. Flex and bison became optional in the same stretch, falling back to a pre-generated parser when they aren't installed, so there's less to set up before building from source.

Benchmark season: SearchBench and the ArangoDB numbers​

We said last month we were calling it benchmark season. Here's the first one.

SearchBench is our open benchmark for search and analytics: 92 queries over OpenTelemetry logs at 100M and 1B records, measuring ingest time, index size and median latency, methodology copied fairly directly from ClickBench. Every adapter is seven shell scripts, so adding an engine is an afternoon.

The first head-to-head is SereneDB against ArangoDB. We started there because ArangoSearch runs on a fork of IResearch, our own search library, taken in 2023. It's the closest thing we'll ever get to measuring ten years of work against its own starting point.

At 100M records on the same machine: ingest and index 20.5x faster, index 5.3x smaller and 16.1x at the median across all 92 queries, 17.2x geometric mean. SereneDB wins 89 of 92. It also answered the full billion with nothing capped and nothing failed, at 46 ms median. ArangoDB's Community Edition caps at 100 GiB of total database size and its 100M index is already 55.3 GiB, so the billion was never on the table for that side.

Which leaves one comparison worth making anyway. Put SereneDB at a billion next to ArangoDB at 100M and SereneDB is still ahead on every task family while carrying ten times the data. Counts land at 20.5 ms against 77 ms, log tailing at 10 ms against 289.5 ms, group-by at 277.5 ms against 1.443 s and joins at 2.1 s against 32.5 s.

Everything is reproducible. Raw per-query results for every engine are at serenedb.com/searchbench. If you think we undertuned something, the adapters are Apache-2.0 and we'd rather take a pull request than defend a guess.

Docs search, powered by SereneDB​

We needed search on our docs. We make a search database. You can see where this went.

docs.serenedb.com and this blog now run hybrid search, BM25 plus vectors, on a SereneDB instance, with streamed AI answers that cite their sources. There's also an MCP endpoint, so an agent can query the docs directly instead of scraping the HTML and guessing.

Then we packaged it, because everyone shipping docs has this problem. Point it at a git repo, a folder, a live website or an S3 bucket and it indexes and re-syncs on its own:

npm install @serenedb/docs-search-react@latest

There's a script-tag embed if you're not on React, a serenedb/docs-search-backend image and a configurator that generates the compose file for you. Run it full-text only or turn on hybrid. AI answers and the MCP server are separate opt-ins.

Docs: Serene Docs Search.

Also shipped​

• You can bind a search condition as a query parameter. TSQUERY is a real value type now, so WHERE col @@ $1 works and an app builds its search query server-side instead of concatenating SQL strings. Prepared statements, arrays of TSQUERY, the operators (||, &&, !!, ##, ^) and 'text'::tokenize('dict') casts all come with it.
• Postgres introspection tells the truth now. pg_depend, pg_rewrite and pg_attrdef were empty stubs, so the ~15 information_schema views that join them silently returned nothing. They're derived from the real dependency graph now and validated byte-for-byte against PostgreSQL 18. Two of them were also handing out oids outside the 32-bit range, which broke any client decoding them as int4. The size functions got the same treatment: pg_relation_size, pg_table_size, pg_total_relation_size, pg_indexes_size, pg_database_size and pg_schema_size return real bytes, where pg_indexes_size used to be hardcoded to zero.
• UNION type end to end: DDL, union_value(), COPY, WAL recovery and use inside inverted indexes.
• Temporal types promoted to first class: TIMESTAMP_S/_MS/_NS, TIMESTAMPTZ_NS, TIMETZ and TIME_NS across the binary wire and index range queries. TIMESTAMPTZ text output follows the session TimeZone the way PG 18 does.
• WAL group commit, with the fsync batched and pipelined across concurrently committing transactions. Commit semantics don't change, only the physical fsync moves. New snapshots stay bounded below the not-yet-durable suffix, so nothing can read a commit a crash could lose.
• sdb_progress, one row per connection with pipeline progress and exact tuple and byte counters for every write statement. pg_stat_activity was an always-empty stub and is now a projection of it, along with every pg_stat_progress_* view. Any query is cancellable with pg_cancel_backend(pid).
• Per-index metrics in sdb_metrics, keyed by relation_id so you can join against pg_class.oid: live docs, segments, files, index size, average commit and consolidation and cleanup times, failure counters.

What to expect in August​

• Sloppy phrase search. A phrase is exact today and the ## gap form pins a fixed distance between two terms. Sloppy phrase gives the whole phrase a slop budget instead, so a query still matches when the text drops a word in the middle.
• Auto-refreshable remote search indices. Today an index over Parquet or Iceberg in a bucket is a snapshot of what was there when you built it. Next month it keeps itself current.
• Logical Postgres replication and triggers. Publications and subscriptions, so an ATTACH'd Postgres can stream changes into SereneDB instead of being re-read.
• Azure. Object storage support beyond S3.
• Faster indexing. More write throughput on top of July's parallel backfill.
• Faster recovery. Less time between starting serened and answering queries after an unclean shutdown.
• More benchmarks. ArangoDB is the first of several. Same rules every time: open adapters, published methodology, raw numbers and the best configuration we can build for every engine on the table.

Kudos​

Some of the most useful input we got in July came from people running SereneDB on their own data and telling us exactly what needs to be improved. A report with a repro is worth as much as a patch and it's usually harder to write.

Thanks to rodion-m for putting inverted indexes under a large code corpus and reporting everything that broke (#970, #971, #972), deadtrickster for a sharp run of reports on vector filtering and BM25 scoring (#961, #962, #964), a-soll for asking about Apple silicon (#883) before SereneDB could build there and emarsden for holding the SQL to the standard (#934, #935). Several of those are fixed in this release. The rest are open with our names on them.

There's also good work in flight we owe review time to: w3lld1 on multi-column search highlights (#932), ivan-digital on an embeddable IResearch-only build target (#976), romanpovol on edge-ngram plus better stemming and normalizing tokenizers (#659) and afigor2701 on pfor encoding (#780).

Want to be in the next one? We tag beginner-friendly work with good first issue, so grab one, ask questions in the issue and we'll get you going.

Everything above is in v26.07.5. Grab it and point it at something. If you'd rather look before you install, the code search demo is live and ⌘K on the docs gets you SereneDB searching SereneDB's documentation. Every raw benchmark result is at serenedb.com/searchbench.

If you like what you see, ⭐ star us on GitHub. Hit a rough edge? Open an issue. The Kudos section above is people who did exactly that.

See you in the next State of Serene.

SereneDB is a database that runs search and analytics in the same query engine. The search half is a C++ library called IResearch that we started in 2016 and still maintain and if that sounds familiar it is because it is what ArangoSearch, a search engine integrated into ArangoDB, runs on.

A billion OpenTelemetry logs, 92 queries and on the other side of the table a ten year old fork of our own code. ArangoDB still ships IResearch, wired into a real database and answering real queries, so for once we get to benchmark ourselves against ourselves. Back in March we published that IResearch won "Search Benchmark, The Game", but that was a microbenchmark and it said nothing about the database around it. Ten years of work either shows up in the numbers or it doesn't and this is how you find out.

Everything below is open so you can rerun it yourself. The head-to-head runs at 100M because that is as far as ArangoDB's Community Edition is licensed to go and past that it stops being a performance question and turns into a licensing one.

TL;DR​

If you only want the numbers.

Results are available at serenedb.com/searchbench and the benchmark itself lives at github.com/serenedb/searchbench.

What SearchBench is​

That March benchmark is the search-benchmark-game, built by the Tantivy team and forked by us to add an IResearch engine. One process, one index, term and phrase and boolean queries, count and top-100 by score. It measures the search library on its own: decode the posting lists, intersect them, score the top 100. There is no SQL layer, no query planner, no columnar storage involved and no joins. So a win there tells you the postings format and the scoring loop are fast and almost nothing about what a database built on top of them does with a real query.

So we built SearchBench, an open benchmark for search and analytics, which we announced at Berlin Buzzwords. Same idea as Search Benchmark, The Game, moved end to end: how fast does the whole database answer a real question.

The methodology is heavily inspired by ClickBench. We run two scales, 100M and 1B log records and measure three things at each: time to ingest and index, size on disk and the median latency across the 92 queries. Result caching is off on both engines. Each engine implements the same handful of shell scripts (install, start, stop, check, load, query, data-size) and a shared driver orchestrates them, so adding an engine is an afternoon.

One rule sits above all of that: every engine gets our best attempt at the best configuration we can build for it. Without that none of these numbers are worth publishing.

The data​

The corpus is generated OpenTelemetry logs. We use the set published by TextBench instead of rolling our own. Every record has 15 columns describing one log message:

The three *Attributes columns are JSON maps. The database has to store all 15 columns, including the ones no query touches. That rule matters more than it sounds, because an engine that only keeps the searchable text will report a beautiful index size and then be useless for actually reading your logs.

The queries​

92 of them, in five groups:

Two of those five are top-100 problems and they differ only in the ordering key: top_k (bm25) ranks by relevance, top_k (time) ranks by timestamp. Keeping them apart matters, because the two get answered by completely different machinery.

Each group is then crossed with how you are asking: single term, conjunction, disjunction, minimum-should-match, phrase, phrase with proximity, prefix, regexp, wildcard, fuzzy, negation and time windows. Plus a term-frequency dimension, because a word in 40% of your logs and a word in 0.001% of them are different problems.

That grid is what produces 92. Every query carries its tags in the result file, so you can slice by group or by filter or by term frequency instead of staring at one aggregate. In the raw files the two top-100 groups are tagged top_k and recent.

Why ArangoDB is the right mirror​

The comparison is worth running because of where the other side comes from. ArangoDB is where an older version of our own code still runs in production and that makes it the closest thing we have to a time machine.

And we do mean our own code. The lineage is traceable in public:

• October 2016. We open iresearch-toolkit/iresearch. That repo is archived today.
• December 2023. ArangoDB forks it into their own org.
• January 2026. They move it out of 3rdParty and copy it into the main tree at lib/iresearch. A little over a million lines in one commit.

What happened in between is what makes this a fair mirror. 51 commits in the two and a half years since the fork and a few of those are merges pulling our upstream work back in. The rest is maintenance: new consolidation defaults, a clang compatibility pass, a memory over-allocation fix, an iterator seek edge case. The same codebase took 1,052 commits in 2017 alone.

So that is a stable dependency doing its job. It also means the engine on the other side of this benchmark is close to the one we handed over, which is what makes it a mirror. Most of what the numbers show is what changed on our side.

How we set it up​

The machine​

Both engines got 100M OpenTelemetry log records on the same machine: a single GCP n2-standard-32, which is 32 vCPUs of Intel Ice Lake at 2.6 GHz and 128 GB of RAM, running Ubuntu 24.04.4 LTS. Storage is a 3.9 TB pd-ssd Google Persistent Disk. One instance each, no cluster and no sharding on either side. SereneDB is v26.07.5 and ArangoDB is version 3.12.9-4, Community Edition.

Community Edition needs a note, because "we benchmarked the free tier" usually means "we benchmarked a crippled build". Not here. Since 3.12.5 the Community Edition ships every Enterprise feature with no time limit. So feature for feature, this is the best ArangoDB there is. What it costs you is the right to run it commercially in production and a 100 GiB ceiling on total database size. That ceiling matters a lot later on.

The data model​

None of the numbers mean anything unless both indexes have the same information to work with. The ArangoSearch view below is the fastest one we managed to build for this workload and every field the queries touch is indexed the same way on both sides. Here is exactly what each engine got.

Tokenization. SereneDB indexes the expression ts_split_by_non_alpha(Body, true), which lowercases and splits on runs of non-alphanumeric characters, over a keyword dictionary with frequency, norm and position enabled. ArangoSearch gets a segmentation analyzer with break=alpha and case=lower and the same three features. Those two are the closest analogs the two engines have. They agree on ordinary log text and they can disagree at the margins, on things like v1.2.3 or user_id, so the term dictionaries are not identical down to the byte. That difference is worth saying out loud and it is nowhere near the size of the gaps below.

Stored columns. SereneDB's index carries all 15 OTel columns as INCLUDE columns, so retrieval, filtering and aggregation run index-only. The ArangoSearch view gets storedValues covering every column the queries return, which is the same deal. Neither engine has to go back to a row store to answer a query.

Scored top-K, meaning the top_k (bm25) family and not top_k (time). SereneDB's index is built with optimize_top_k = 'bm25(1.2, 0.75)', our own WAND implementation inside IResearch. The ArangoSearch view is created with optimizeTopK enabled and the matching BM25 parameters. Both sides get to skip blocks they can prove cannot make the top of the list.

Query translation. Every SereneDB query has a hand-written AQL twin with the same filters, the same windows, the same grouping, the same sort and the same limit. The mapping is mechanical and both sides are documented: SereneDB full-text functions against ArangoSearch AQL functions.

That last row is the one place we had to improvise. ArangoDB has no regular-expression matcher inside SEARCH at all, so the nine queries that use one are hand-translated into whichever primitive comes closest: STARTS_WITH(d.Body, 'charg') where the pattern is a prefix followed by .*, LIKE(d.Body, 'c_che') where a . sits inside the word. We could have marked those queries unsupported and moved on. Giving ArangoDB the nearest primitive instead hands it an advantage, because STARTS_WITH walks one contiguous range of the term dictionary while the regex it stands in for compiles to an automaton that can walk all of it. Nine of the 92 queries are tilted ArangoDB's way and we would rather say so up front.

Both query files are in the repo. If a translation looks unfair the diff is right there and we want to hear about it.

How we measure​

Timing. Three runs per query and we report the best of the last two. Between queries we stop the engine, drop the OS page cache, start it again and wait for it to report healthy. Each measurement is the client round-trip for that one query, taken from psql's \timing on the Postgres-wire side and curl's %{time_total} on the HTTP side, so the cost of spawning and connecting a client stays out of the number.

The 60 second rule. Any single query gets 60 seconds. If it does not finish, we record that it did not finish. There are several of those below and they are all on the ArangoDB side, so we want to be precise about what it means: it is not a latency of 60 seconds, it is a query that never came back.

What this does not measure. The driver runs one query at a time from a single client, so everything here is latency and none of it is throughput. There is no concurrent load, no ingest running alongside queries, no updates or deletes and nothing distributed. Memory footprint is not recorded either. Those are all fair questions about a log store and none of them are answered here.

Load and size​

Before any query runs, there is the part where you wait.

Ingest is 20.5x faster and the index is 5.3x smaller. Both numbers cover the same work: read the corpus, tokenize the body, build the inverted index, store every column for retrieval.

The ingest number needs one note, because the two sides do not get the data exactly the same way. Both engines run in Docker with --network host. SereneDB builds the index straight off the parquet: the table is a view over read_parquet and CREATE INDEX consumes it in process, so nothing gets serialized and nothing crosses a socket. ArangoDB cannot read parquet at all, so the corpus goes through a throwaway serened container that emits NDJSON on stdout, arangoimport reads that over a pipe at its own default parallelism and inserts the documents and only then does the view link index them. Some of those 18.6 minutes is JSON serialization and JSONL parsing rather than index building. We report the pipeline because it is the pipeline you would actually have to run, but if you want the pure indexing cost on the ArangoDB side it might actually be lower.

Query latency​

Median query latency per task family.

2 of the 9 join queries never came back. They hit our 60 second ceiling and we record them as 60 seconds, which understates ArangoDB's real time and therefore understates the gap. Both sit above the join median of 32.5 s, so neither is what that bar is measuring. The other seven finish between 13 and 45 seconds. Drop the two capped queries and the median across the remaining 90 is 15.6x, against 16.1x for all 92.

16.1x at the median, 17.2x geometric mean, across all 92 queries. Our best single query is 460x: a time-windowed top-100 that takes us 8 ms and ArangoDB 3.7 s. We quote the median instead because that is the number that tells you what the database feels like to use. It is also the number that survives an argument.

SereneDB is faster on 89 of the 92 queries. On the three it loses, both engines land somewhere between 1 ms and 15 ms, close enough to the floor that run-to-run variance decides the winner rather than anything about the engines.

The billion logs benchmark​

We ran the full billion on our side. SereneDB answers all 92 queries at that scale with nothing capped and nothing failed.

Ingest and index took 8.5 minutes and the index is 120 GiB on disk.

None of it scales linearly. Ten times the data costs a lot less than ten times as much:

Latency is the one that matters. Take each query on its own and the median slowdown is 2.8x for ten times the corpus, with a geometric mean of 2.9x. Build time came in slightly under linear at 9.4x. Size went the other way at 11.5x, so the index gets proportionally fatter as the corpus grows. That one is moving against us and we would rather say so than let someone find it.

Why we can't run ArangoDB at a billion​

ArangoDB's Community Edition enforces a 100 GiB total database size limit. Go over it and you get warnings for two days, then the deployment drops into read-only for two days, then it shuts down. At 100M logs the ArangoDB instance is already 55.3 GiB, so the licensed room runs out somewhere around 181M logs. A billion would have put us far past that, so we did not run it.

That leaves us extrapolating the two things that scale predictably. Straight-line from the measured 100M run, a 1B ArangoSearch index comes to roughly 553 GiB and at least 3.1 hours to build. Both are floors if ArangoSearch scales the way ours does: our own index grew 11.5x for ten times the data. Read the same figures as capacity and one 100 GiB instance holds around 181M logs on ArangoDB against around 830M on SereneDB. If you want measurements instead of our arithmetic, SearchBench is open and reproducible: the ArangoDB adapter is in the repo, so anyone holding a license that permits it can run the billion and publish what they get.

Query latency is the one thing we cannot extrapolate, so we did the next best thing and lined our 1B numbers up against ArangoDB's 100M ones, with only one side carrying ten times the data.

So those are the numbers. The rest of this post is about where they come from.

Where the speed comes from​

Query categories explained​

Inside the search engine​

We reworked most of IResearch: the retrieval path, the scoring loop and the postings format itself. The details are published as the Search optimization journey series and each part maps onto something in the table above.

• Collecting top-K candidates, part 1. The pruning threshold has to be exact after every insert, because a stale bound makes every later block do more work than it needs to. Tantivy hit the same thing and swapped in a binary heap for exactly that reason.
• Block scoring, part 2. Score a whole block of postings at a time instead of a document at a time. Tantivy is now exploring the same approach, citing that post.
• Optimize norm gathering, part 3. Norms sit on the hot path of every scored query, so how you fetch them shows up everywhere in top_k (bm25).
• How to efficiently execute two phase queries, part 4. Matching separated from scoring, so the expensive half only runs on the documents that survive the cheap half.
• Adaptive posting list format, part 5. The encoding adapts to the density of each term rather than forcing one layout on everything, which is most of why the index is 5.3x smaller.
• A new columnar store. This is the one that never got a post and it matters twice over. The store holds the storedValues that retrieval reads and the norms that BM25 scoring reads and in the version ArangoSearch runs it does no compression at all: norms are created with compression::none hard-coded. So the same bytes are bigger on disk and more expensive to pull through the scoring loop. Ours compresses, which is a good part of both the 5.3x size difference and the top_k (bm25) numbers.
• BlockMax MaxScore, not written up yet. For scored disjunctions, per-block score bounds let the iterator skip whole blocks that cannot reach the current top-K threshold. That is what carries the or entries in top_k (bm25).
• A filter optimizer. Query rewriting used to happen inline while a filter was being prepared. It is now a rule registry with separate rule families for booleans, negation, ranges, terms, lowering and levenshtein prefixes and prepare does nothing but look up cookies. Wildcards and regexes lower to an AutomatonFilter and fuzzy matches to a LevenshteinAutomatonFilter, so regexp, like and fuzzy all end up as automaton intersections against the term dictionary instead of separate hand-written code paths.
• Parallel per-segment prepare. Preparing a filter is not free and for a fuzzy query it is most of the query: before anything can match, the engine has to walk the term dictionary of every segment collecting the terms within edit distance. That work now runs per segment in parallel.
• A dedicated path for counting. Counting is a first-class operation on the iterator tree rather than something layered on top. count() is pure virtual on DocIterator, so every iterator implements it and a conjunction or a disjunction can report how many documents it matches without ever scoring or collecting them. Thirteen iterators implement it today. On top of that the planner pushes an empty projection into the scan for count(*) shapes, so the column store is never opened either. The plan says so out loud:

╭─ IRESEARCH_SCAN ─────────────────────────────╮
│ Index: logs_idx                              │
│ Index Filter:                                │
│ ╭─ Term ───────────────────────────────────╮ │
│ │ Field: ts_split_by_non_alpha(body, true) │ │
│ │ Value: payment                           │ │
│ ╰──────────────────────────────────────────╯ │
│ Output: row-count only                       │
╰──────────────────────────────────────────────╯

Around the search engine​

• Parallel execution. ArangoSearch exposes a parallelism option on SEARCH that spreads index segments across threads, so we tried it. The workload came out around 1.5x slower, so the published run leaves it at the default. On our side both halves run in parallel: preparation per segment inside the search engine, then execution out here in the query engine.
• Vectorized execution. Operators run over batches of values. Aggregating a few million matched rows by service or by minute is a columnar problem where the index plays no part and the group_by family is essentially a measurement of that.
• Join strategies with a cost-based optimizer. Correlating two services through a trace id is a join. For us it becomes a hash join inside the vectorized engine, running over columns the search index already holds, with a planner picking between several join algorithms. In ArangoDB the same question becomes a nested loop: for every log on the left, go look up the matching logs on the right. At 100M records that difference is architectural and no amount of postings tuning closes it.
• Regular expressions. ArangoSearch has no regex matcher inside SEARCH at all, so the nine regexp queries had to be hand-translated into STARTS_WITH and LIKE. We have one thanks to aksel2904, who contributed the regexp filter.

Don't trust us, run it yourself​

Everything behind every number above is in github.com/serenedb/searchbench: the adapters, the 92 tagged queries in each dialect, the shared driver and the raw result JSONs. Each engine directory has a README listing its env vars.

git clone https://github.com/serenedb/searchbench
cd searchbench/arangodb   # or serenedb
SEARCHBENCH_DATASET=otel_logs_100m SEARCHBENCH_DATA_DIR=/data ./benchmark.sh --index

--index fetches the corpus, loads it and builds the index. Drop it to re-run queries against an engine that is already loaded. Omit SEARCHBENCH_DATASET and you get the billion. There is a 1M smoke scale that streams about 50 MB over HTTPS if you want to check an adapter before committing to a real run.

Full per-query results for every engine we have wired up, including the ones absent from this post, are on the results page.

Send us a better run​

If you think we got something wrong or that we left an engine undertuned, the fix is a pull request. We mean that literally. Every adapter in that repo is somebody's best attempt at configuring a database they did not write and ours are no exception. If you know ArangoDB better than we do, we would rather learn it from your PR than defend a number we got by guessing.

The same goes for the workload. If a translation looks unfair, if a query family is missing, if the corpus looks nothing like your logs, open an issue and say so. New engines are welcome too and cost about seven small scripts.

A benchmark is only worth anything when the people being measured can push back on it. That is why this one is Apache 2.0 with no conditions attached to publishing results and why the queries and the adapters and the raw JSONs are all sitting there in the open. Everyone in this field is guessing about everyone else's engine and the only way out of that is to compare notes properly. We would like SearchBench to be where that happens.

Conclusion​

Ten years on, the engine we started in 2016 as a pet project is unrecognizable from the one still shipping inside ArangoDB and this is the first time we have put numbers on that.

Here is what a billion OpenTelemetry logs look like on a single machine. Ingest and index takes eight and a half minutes and lands at 120 GiB on disk. Of the 92 queries, the 83 non-join ones come back at a median of 29 ms, with just over seven in ten under 100 ms. The nine expensive joins that correlate two services through a trace id take just a few seconds.

If you are running search workloads on something that makes you choose between full-text and analytics, SereneDB is worth twenty minutes of your evening. It speaks the Postgres wire protocol, so whatever client you already have will do and every number in this post is a CREATE INDEX away from being reproducible on your own dataset.

This is the first of these and not the last. Several engines are already wired up with results in the repo at both scales, so the next comparisons are mostly a matter of writing them up.

If you find this interesting, we'd be grateful if you support SereneDB with a star on GitHub. For an early-stage project, it means more than you might think.

Welcome to The State of Serene, our monthly note on what we shipped, what we learned and where SereneDB is heading. This is the first issue, so before we get to June, let's answer the obvious question: what is SereneDB and what can you actually do with it today?

Instead of moving your data to the model, point the model at the data. Imagine an agent gets asked: "Which enterprise accounts are unhappy about latency and how much revenue is at risk?" It can't grep 100GB of tickets and it won't fit them in a context window, so it asks SereneDB instead. Search pulls the tickets that are really about slowness, even the ones that never use the word "latency". The same query joins them to accounts and sums the revenue at risk by plan. The agent gets back a handful of numbers it can act on, not a stack of tickets to read.

So an agent needs two things: the most relevant data and the statistics over it. Hybrid search gives the relevant part. Analytics gives the numbers on top. SereneDB does both in one query and hands back a small answer, so the agent spends its tokens on the decision, not on scanning files.

That's SereneDB: lakehouse search and analytics for your agents. One Apache-2.0 database that speaks Postgres and reads your data where it already lives, whether that's Parquet and Iceberg in the lake, your Postgres or your warehouse. It indexes that data in place and hands back the slice that matters, so nothing gets copied into another system and nothing goes stale. Think of it as a compute and delivery machine for your agents.

What you can do today​

It's Postgres-compatible​

Compatible on two levels, the wire protocol and the SQL dialect, so most of what you already use just works.

• Connect with psql, any Postgres driver or your favorite ORM. Nothing new to learn.
• Standard types plus pg_catalog and system tables, so BI tools and ORMs introspect it like any Postgres.
• See what's supported and the few quirks in the compatibility docs.

Search and analytics in one engine​

One inverted index holds full-text and vectors, sitting right next to columnar, vectorized analytics. Same data, same query.

• Full-text search: BM25, phrase, fuzzy, wildcard and regex, plus a stack of tokenizers.
• Vector search and hybrid that fuses both with tunable ranking.
• Geospatial search in the same index.
• Real analytical SQL right beside it: joins, CTEs, window functions and GROUP BY over the same rows. Filter by relevance, then aggregate, in one query.
• It all runs on iresearch, our own open-source engine and a fast one.

Index your data where it lives​

Point SereneDB at your data and query it like a normal table, no ETL and no copies.

• Remote files: Parquet, JSON, CSV and Iceberg on S3, HTTP or Hugging Face.
• ATTACH a live Postgres database and join it against your indexed data.
• Bring it in for one source of truth or leave it where it is and index it in place. Same SQL either way.

See it end to end in the code search demo: hybrid search over 11.4M rows read straight from Hugging Face. More on that below.

What happened in June​

A few of June's highlights worth your time.

Search queries prepare in parallel​

Setting up a search query used to walk the index segments one at a time. Now the prepare phase and the scan both fan out across segments, so query setup scales with your cores instead of pinning a single one. Ranked queries also skip documents that can't reach the top n. Under real concurrency it adds up. Here are Top-K queries at 32 threads, before and after this month's search work:

Every one of the 26 filter types came out ahead, from 1.06x (gt) to 2.48x (ngram).

And it scales with cores. BM25 scorer streaming runs 1.90x faster at 32 threads and 2.08x at 64. Top-K gets most of its win by 32 threads.

Smarter search filters​

Search filters now run through a set of optimizer rules that lower each one to the cheapest form that still matches, done once up front instead of ad-hoc code in the prepare path. Some of the rewrites:

Cheaper filters and the lowering runs once at optimize time instead of per query.

A rebuilt server that speaks two protocols​

We rebuilt the server from the ground up as a sans-io network server with zero-copy framing. Two protocols talk to it:

• Postgres wire protocol: a clean implementation with auth, a real CLI and a psql-style shell out of the box, plus background schedulers for maintenance.
• Elasticsearch HTTP API: a drop-in _bulk, _search and scroll, so existing Elasticsearch clients point straight at it. There's a migration guide for when you want to move queries to SQL.

Underneath, we also started collapsing the storage layer, so fewer moving parts and a simpler single binary.

And it's faster. A wire bench (trivial queries, pgbench-style, 16 clients), so it measures the server and protocol layer, not query execution, since the engine underneath is unchanged. select1 is a bare SELECT 1 (pure transport cost), wide is a parse-heavy SELECT 1 WHERE 1=1 AND ... AND 10=10 (isolates parse cost) and simple/extended/prepared are the Postgres protocol modes. New server against the old one:

The demo that ties it together​

We put the whole thing in a live demo: code search over ~11.4M Codeforces solutions and their problem statements, read straight from Hugging Face with no ingestion job. Solutions get GitHub-style code search (exact and fuzzy), matched with a new sparse ngram tokenizer. Tasks get hybrid keyword plus semantic. And there's an MCP endpoint so an agent can search it directly.

The fun part: the backend took about 15 minutes. We handed Claude our docs and it wrote the whole thing, pointing the database at the remote data, building the indexes and writing the queries.

Go try it: codesearch.serenedb.com · repo: github.com/serenedb/codesearch · writeup: Code search over a data lake.

Also shipped​

• a new on-disk index format, the base layer a lot of the above sits on
• column statistics from the index, so aggregates like SUM hit faster paths (6-7 ms, was 23-27 on 25M rows)
• VARIANT support (store, load, shredding) and array indexing for semi-structured data
• the inet type
• nanosecond timestamptz

What to expect in July​

On the engineering side​

• landing ts_dict (analytics from the search index)
• a new vector-search design we've been cooking
• more efficient columnar storage and execution, with dynamic filters
• indexing of complex types, including schema-agnostic indexing
• foreign data sources via CREATE SERVER, including a ClickHouse connector
• ongoing work on logical replication (publications and subscriptions)
• PG-compatible authorization and RBAC

Benchmark season​

Summer's here, so we're calling it benchmark season. Over the coming weeks we're putting SereneDB up against the best-in-class databases in the open, with methodology and raw numbers published like we always do. First up is a billion-record benchmark on search-heavy workloads, our biggest test so far. Stay tuned.

Kudos​

A lot of SereneDB's search surface comes from external contributors. A few we want to shout out:

• romanpovol built the path-hierarchy and pattern tokenizers
• afigor2701 worked on inverted-index internals (WAND data handling and integer encoding)
• aksel2904 added the regexp filter
• abhishek593 added the union tokenizer
• k-morozov did the synonym tokenizer with WordNet support, plus the CREATE PUBLICATION / CREATE SUBSCRIPTION grammar
• hinata34 co-authored the filter optimizer

Thanks folks, keep the PRs coming.

Want to be in the next one? We love outside contributions and we're happy to help you land your first PR. We tag beginner-friendly work with good first issue, so grab one, ask questions in the issue and we'll get you going. There's plenty of things to dig into.

If "search and analytics on the data where it lives" sounds useful, the fastest path is the code search demo. And if you like what you see, ⭐ star us on GitHub. Hit a rough edge or have a use case? We'd love to hear it: open an issue or just say hi.

See you in the next State of Serene.

Most data never moves. It just sits as files in object storage: Parquet and JSON in S3, datasets on Hugging Face. A lot of it, in open formats. That's a data lake: cheap to store, but you can't search it on its own.

Search and analytics tools can't read the lake directly; they need their own copy of the data. So you set up a separate system, write code to copy the data into it, keep that copy up to date when the source changes and add a warehouse on the side for the numbers. That's a lot of extra work, just to move data you'd rather leave where it is.

SereneDB lets you skip that. You point it at the files where they are, build an index and search them in place. Because the index also holds your columns, one query can both search and compute statistics, and the data never leaves storage. And it isn't only for code search: the same view-and-index steps work on any Parquet in any bucket.

To show what that looks like we built a code-search system on it, in about 15 minutes: roughly 14,000 programming problems and 11.5 million accepted solutions, about 14 GB of text in all, searchable by keyword, by code substring and by meaning, with live statistics over whatever you searched for. The data is Parquet on Hugging Face; we put SQL views over the files and built indexes on the views. SereneDB speaks the Postgres wire protocol, so psql and any Postgres client just work.

Index remote sources, store only the index​

A view is just a SQL definition. Ours read Parquet straight off Hugging Face, reshaping each source into a common shape and gluing different sources together with UNION ALL:

CREATE VIEW solutions_v AS
  -- Codeforces: ~11.4M accepted submissions, with a natural numeric id
  SELECT submission_id::BIGINT      AS id, problem_id AS task_id, source AS code, ...
  FROM read_parquet('hf://datasets/open-r1/codeforces-submissions/...')
UNION ALL
  -- MBPP reference solutions (no numeric id -> hash the natural key)
  SELECT hash('mbpp/' || task_id)   AS id, 'mbpp/' || task_id AS task_id, code, ...
  FROM read_parquet('hf://datasets/google-research-datasets/mbpp/...')
UNION ALL
  -- HumanEval reference solutions
  SELECT hash('he/' || task_id)     AS id, task_id, prompt || canonical_solution AS code, ...
  FROM read_parquet('hf://datasets/openai/openai_humaneval/...')
UNION ALL
  -- Rosetta Code: per-language source
  SELECT hash('rc/' || task_name)   AS id, task_name AS task_id, code, ...
  FROM read_parquet('hf://datasets/christopher/rosetta-code/...');

CREATE INDEX solutions_idx ON solutions_v
  USING inverted(id, task_id, lang, code code_grams);

solutions_v unions four sources this way; tasks_v unions six. That's seven distinct Hugging Face datasets across the two indexes, each with its own schema, flattened into one searchable relation.

That's the whole ingest. No table is created and nothing is downloaded — there's no wget, no COPY, no "first pull it onto a box" step. The view reads the remote Parquet in place (pulling only the columns and rows it touches), and the index build streams from there. The single thing we store is the index. Adding a source is one more SELECT, and keeping the index current as those files change is what we're building next.

So there's no search stack to stand up: no Elasticsearch, no ETL job, no copy to keep in sync. You point SereneDB at data already sitting in S3, write CREATE INDEX, and you have search.

Keep columns in the index with INCLUDE​

An inverted index maps each term to the documents that contain it: a posting list of doc ids. By default a match hands you those ids, and you'd look the rest of each row up in the source. INCLUDE(...) changes that: the listed columns are stored in the index, right next to the postings, so a result comes back whole, with no lookup into the source.

CREATE INDEX solutions_idx ON solutions_v
  USING inverted(id, task_id, lang, code code_grams)
  INCLUDE (id, task_id, code, code_len, lang, time_ms, memory_kb);