We keep putting SereneDB next to other databases and search engines and publishing whatever falls out. This is the fourth one. The three before it:

ArangoDB, interesting because SereneDB is built by the people who wrote ArangoSearch, so it measured us against our own earlier work.

The Postgres family, ParadeDB, TigerData and Postgres, interesting because SereneDB is Postgres compatible and we wanted to know what we actually add to that ecosystem.

The Lucene family, Elasticsearch, OpenSearch and CrateDB, interesting because Lucene is the most robust search library there is and it powers the de facto industry standard search servers.

Today we come at it from a different angle. ClickHouse is known as a very fast real-time OLAP database and not as a search engine. That is how the market reads them and the reading is fair, but they have been improving their inverted index lately. They became core infrastructure for observability, observability means tons of logs and searching logs means searching unstructured data. Text search is the obvious next step for them.

TL;DR​

As usual numbers first. Ten billion OpenTelemetry logs, 92 queries, 60 second cap, one instance, hot median latency:

SereneDB answers all 92 queries at all three scales, 100M, 1B and 10B, with nothing unsupported and nothing over the 60 second cap. No other engine in this series has done that.

We run ClickHouse twice, with and without TextBench's ORDER BY (ServiceName, Timestamp) sorting key. It is worth more than the text index here and that is the catch: a sorting key only speeds up the queries you know in advance, like most of ours filter on a service or a time window, so it works well. Real search traffic might not work that way, because people and agents ask for whatever they want. SereneDB table has no sorting key at all.

Full results are at serenedb.com/searchbench and the benchmark itself lives at github.com/serenedb/searchbench.

Chapter 1. Where SearchBench came from​

In April this article caught our attention and we liked the idea a lot. ClickHouse had built a small version of ClickBench for log analytics and called it TextBench. It is a proper observability use case on the OpenTelemetry schema and the dataset is a good one, billions of logs ingested from parquet files. The weak part was the query set, a handful of filtering queries showcasing the brand new inverted index.

So we took it as inspiration and pushed it further. Same dataset, many more queries, covering a good chunk of what Elasticsearch can actually do. We added relevance ranking, faceting and joins on top. The result is SearchBench, an open benchmark for search and analytics built the way ClickBench is built.

ClickHouse's TextBench numbers were excellent. Plenty of search functionality is still missing on their side and they are still the fastest thing we have put ourselves against so far, so this one was going to be interesting. For it we extended the dataset to 10 billion logs on top of the usual 100M and 1B runs. With the other engines in this series that would have meant waiting the whole day. ClickHouse is quick enough that it is an hour. They also added phrase search to their index recently, which makes the comparison better still, because storing positions next to the postings efficiently is a genuinely hard problem.

Chapter 2. What SearchBench measures​

We announced SearchBench at Berlin Buzzwords and first published it in July when we ran it against ArangoDB. Following ClickBench, every engine implements the same seven shell scripts and a shared driver orchestrates them, so adding an engine is an afternoon of work.

This time we run three scales, 100M, 1B and 10B log records and measure three things at each: time to ingest and index, size on disk and the median latency across the 92 queries. Result caching is off everywhere.

Timing. Three runs per query, we report the best of the last two. Each measurement is the client round trip for one query, taken from psql's \timing on SereneDB and from the ClickHouse client on ClickHouse. We deliberately do not use the server's own reported timing, because that hides serialization and transfer.

Cold and hot. At 1B and 10B the driver stops the engine, drops the page cache and starts it again before every single query, so run one is a genuine cold pass and runs two and three are warm. At 100M it does not, so treat the 100M cold column as a warm-ish number. This matters a lot at ten billion and Chapter 7 is mostly about it.

The 60 second rule. Any single query gets 60 seconds. If it does not come back in that window we record that it did not finish. There are plenty of those below. It means the query was still running at 60 seconds and we killed it.

What this does not measure. One query at a time from a single client, so everything here is latency. Concurrency, updates, distribution and memory are all fair questions we do not answer.

We give every engine the strongest configuration we know how to build. We will have got some of it wrong, which is why every adapter sits in the repo where you can check it.

The OpenTelemetry corpus​

The corpus is generated OpenTelemetry logs. We use the set published by TextBench instead of rolling our own. Every record has 15 columns describing one log message:

The three *Attributes columns are JSON maps. The database has to store all 15 columns, including the ones no query touches. An engine that keeps only the searchable text reports a beautiful index size and is then useless for reading your logs.

Ten billion records is ten parquet parts of roughly 40 GB each, so about 400 GB going in.

The 92 queries​

92 of them, in five families:

Two of those five are top 100 problems and they differ only in the ordering key. top_k (bm25) ranks by relevance and top_k (time) ranks by timestamp. They get answered by completely different machinery.

Each family is then crossed with how you are asking: single term, conjunction, disjunction, minimum should match, phrase, phrase with proximity, prefix, regexp, wildcard, fuzzy, negation and time windows. Then we cross that again with how common the word is, because searching for something that shows up in 40% of your logs is a completely different job from finding one that shows up in 0.001%.

Chapter 3. How we configured each engine​

The machine​

One GCP n2-standard-32, so 32 vCPUs of Intel Ice Lake at 2.6 GHz and 128 GB of RAM, running Ubuntu 24.04.4 LTS on a 3.9 TB pd-ssd. One instance of each engine, no clustering and no sharding, at every scale including ten billion.

What each engine got​

Every configuration stores all 15 columns and indexes exactly one text field. Body is tokenized the same way everywhere: split on runs of whitespace, punctuation and symbols, then lowercase, with no stemming and no stopword removal. Here is where they differ.

We run ClickHouse two ways, because the sorting key turned out to matter more than the text index. ORDER BY (ServiceName, Timestamp) is TextBench's own key and it physically clusters the rows by the two columns most of the 92 queries filter or order on, so ClickHouse gets to skip whole granules on a service filter and read a time window more or less sequentially. ORDER BY tuple() is the same table with that switched off. We ran both because otherwise you cannot tell what the text index is doing from what the storage layout is doing for it. Chapter 4 onward keeps them apart.

SereneDB has no sorting key at all. So rows land in whatever order the parquet readers produce them and nothing is clustered on anything and every number below is an unsorted SereneDB against a sorted ClickHouse.

Phrase search is on and it is not free. support_phrase_search = 1 makes the text index store positions and the adapter README measures what that costs at 100M. Phrase queries come back 50 to 90% faster, paid for with 2.5 times the index size and 25% more ingest time. We turned it on anyway, because a log search engine that cannot do phrases is not really a log search engine and because this is the part we especially wanted to test.

regexp, prefix, wildcard and fuzzy are full scans, ClickHouse doesn't optimize them. They run as arrayExists predicates over tokens(lower(Body), 'splitByNonAlpha'), which matches the index tokenizer exactly, so the answers are right. They are also roughly 100 times slower than hasToken. We put a two phase filter in front of every one of them to get back what we could, which Chapter 6 goes through.

Chapter 4. Load time and index size​

SereneDB ingests faster than either ClickHouse configuration at every scale and end up in about two thirds of the space. On load time this is still the closest anyone has come to us.

Sorted ClickHouse takes 1.41x, 1.29x and 1.34x our time across the three scales. The three Lucene engines sat between 21x and 53x. Both of us read parquet in process, which is most of that difference.

The two ClickHouse rows price the sorting key. It costs 8.7% more ingest time at a hundred million, 1.3% at a billion and 5.6% at ten billion and it saves 12% on disk at all three. The sorted table ends up smaller even though it does more work during ingest.

On disk sorted ClickHouse is 1.47x, 1.49x and 1.50x our size and that ratio barely moves across two orders of magnitude. Unsorted it is 1.68x, 1.70x and 1.70x. Clustering by service and time makes the low cardinality columns compress far better and it is still half again our footprint.

Chapter 5. Ninety two queries at 100M, 1B and 10B​

One scale at a time, median hot latency per query family. The gap widens at every step.

100 million logs​

A bar marked n/a is a family the engine has no syntax for, so there is nothing to draw. ClickHouse's text index does boolean token matching with no ranking, so it cannot run the BM25 top-k family at all.

This is the scale where ClickHouse looks closest and where the sorting key pays off most. Both configurations express the same 70 queries and neither times out. Sorted, its median on those 70 is 29.0 ms. Unsorted, 44.0 ms. Ours on the same 70 is 12.5 ms and across all 92, including the 22 ClickHouse has no syntax for, 15.0 ms. We answer every one of them.

So the sorting key is worth 1.52x here and the families it helps most are the ones that touch ServiceName: grouping goes from 48 ms to 24.5 ms and the joins from 320 ms to 169 ms. At a hundred million rows the table is small enough that clustering buys most of what a search index would.

One billion logs​

Ten times the data and the sorting key is no longer enough. Sorted ClickHouse goes to 349.0 ms on the 70 it answers and unsorted to 482.0 ms on 69, against our 26.5 ms on the same subset. We again answer all 92, at a median of 31.0 ms. SereneDB is faster on 65 of the 70 queries sorted ClickHouse finished and on all 69 the unsorted one finished.

Sorting is still worth 1.38x overall and 2.19x on the joins, 5.32 s down to 2.43 s. It is the only reason the sorted table gets through all 70 without a timeout while the unsorted one loses a fuzzy count.

Give ClickHouse credit for the five it keeps. Four are the same ServiceName group-bys it won at 100M and one is a join, so sorting is still doing real work. It just stopped being enough.

Counting is the worst of it, 384.0 ms against 18.0 ms, because more than a third of the count family runs as a scan and a scan over a billion rows costs what a scan over a billion rows costs. Log tailing is the other one, 151.5 ms against 10.5 ms and that is the family ClickHouse's sorting key was supposed to own, because ordering by Timestamp exists so you can read the newest records without searching for them. That gap gets much worse at the next scale.

Ten billion logs​

This is where ClickHouse comes apart. Sorted gets 60 of 92 back at a median of 1912.5 ms, unsorted 48 of 92 at 2301.0 ms. We answer all 92 at 173.5 ms.

Every ClickHouse bar here is a median over what that configuration finished in the family, so the sorted one covers 27 of 32 counts, 13 of 14 group-bys, all 16 tail queries and 4 of 9 joins. Ours cover all 92. The n/a bar is the missing BM25 family.

The joins need care. We finished nine, sorted ClickHouse finished four and beats us on those four at 20.8 seconds against our 39.6, unsorted finished three and loses on those three at 47.7 seconds against our 38.2. Every one of these joins probes on ServiceName, the leading column the sorted table clusters by, which is also why the ones it drops are the ones probing a different service.

On the 60 both engines finished, sorted ClickHouse is at 1912.5 ms and we are at 102.5 ms. We are faster on 52 of those 60.

What it still wins is the ServiceName group-bys it has won at every scale, plus those joins. Grouping is the family where it is closest, 2.61 s to our 1.29 s.

Log tailing is where the gap is widest. Sixteen queries asking for the newest hundred matching records, 433 ms on sorted ClickHouse against 12.5 ms on SereneDB. ClickHouse sorts its table by Timestamp to answer exactly this shape. Ours is not sorted by anything.

Our line is flat at 92. Nothing unsupported and nothing timed out at any of the three scales. The corpus grew a hundredfold and every query still came back.

ClickHouse barely times out at the two smaller scales, so its 70 is almost purely the 22 it cannot express. Ten billion is where it comes apart: the sorted table loses 10 more to the clock and the unsorted one loses 22, taking them to 60 and 48.

Thirteen of the ones it did finish at ten billion nearly did not. Their first run takes 60 to 105 seconds, well past the cap and then the same query comes back in 20 to 53 seconds once the pages are in memory. Give up after the first slow attempt and you write down thirteen timeouts that are not timeouts. If you are benchmarking anything big enough for the page cache to matter, keep the cold run separate from the warm ones. Otherwise you are measuring your disk and calling it a query.

How SereneDB scales from 100M to 10B​

A hundred times the data costs SereneDB 11.6x on the overall median. That average hides a huge spread:

Log tailing is flat. A hundred times the corpus costs 2.5 milliseconds more, because finding the newest hundred matching records never touches the rest. Counting and ranking grow well under linear too.

Chapter 6. What ClickHouse's text index cannot do yet​

ClickHouse does not support 22 of the 92 queries. There is no way to write them against its text index, so both configurations leave the same 22 blank at every scale. Sorting changes how fast the other 70 run and changes nothing here. Here is the whole surface, query by query.

Every SereneDB form on the right is served by the inverted index, so the column to read is the middle one.

Relevance is the big one. No BM25 means 21 of 92 queries are absent and the scored top-k family has no ClickHouse column at any scale. Add the proximity query and that is the full 22. This is where the text index is today, but ClickHouse only started on this recently and they are moving fast.

The scan-backed shapes are still the first to run out of time. Sorting cleaned up most of the 10B timeouts and the ones left over are all the same shape. Strip out the joins and sorted ClickHouse has five timeouts at ten billion. Every single one is a scan-backed shape: four fuzzy and one suffix wildcard. Nothing that the text index can actually serve ran out of time. Sorting by (ServiceName, Timestamp) does nothing for a predicate that has to look inside Body on every row, which is why these are the ones left standing.

The cost is visible well before the cap too. The seven regexp, prefix and wildcard counts that do finish land between 29.5 and 53.4 seconds warm. We answer the same seven in 20 to 54 milliseconds, because a term dictionary turns all of them into a lookup.

Positions do work and at this scale they really show. Phrase search is genuinely index-backed and the 50 to 90% improvement the adapter measured is real. All nine phrase-shaped queries it can express finish at ten billion on the sorted table and none of them come close to the cap. That is the part of the text index we most wanted to stress and it held.

We did our best to speed these up. Every scan-backed shape runs in two phases: a cheap substring check first, then the exact check on whatever survives. We wrote that for ClickHouse and their numbers above would be worse without it.

Chapter 7. What breaks at ten billion logs​

Ten billion rows on one box broke two things on our side before it produced a number. Nobody publishes this part of a benchmark, so here it is.

The joins had to be rewritten​

All nine joins used to be count(DISTINCT a.TraceId) over an inner self-join. That form materializes every matching pair before it aggregates anything, planning as HASH_JOIN into a projection into an ungrouped aggregate. At a hundred million and at a billion it is fine. At ten billion it was simply too slow.

They now deduplicate before the join instead of after:

SELECT count(*) FROM (
    SELECT DISTINCT TraceId FROM otel_logs_idx
    WHERE TraceId <> '' AND ServiceName = 'frontend' AND Body @@ 'failed'
) a
WHERE EXISTS (
    SELECT 1 FROM otel_logs_idx b
    WHERE b.TraceId = a.TraceId AND b.ServiceName = 'payment'
);

Same answers and it plans as HASH_GROUP_BY into a RIGHT_SEMI HASH_JOIN into a count, so the fan-out never gets built. It is a win at the smaller scales too.

We tried to do the same trick on ClickHouse and it came out slower, so we left it on the shape its own planner handles best. That is the rule we have followed for every adapter in the suite.

The numbers settle it. Put our unsorted table next to the ClickHouse that stores its rows the same way and we take the join family at all three scales: 162 ms against 320 ms at a hundred million, 1.57 s against 5.32 s at a billion and 38.2 s against 47.7 s at ten billion. Each engine runs the form its own planner likes and we are quicker every time.

Sorted ClickHouse is the only configuration that beats us here, taking four of the nine at ten billion at 20.8 seconds against our 39.6. That is the sorting key and nothing else. Every one of these joins probes on ServiceName, the leading column it clusters by, which is also why the ones it drops are the ones probing a different service. Joins at ten billion are still slow and they are ours to speed up.

Cold reads are a different regime​

At 1B and 10B the driver drops the page cache before every query, so the first run genuinely touches disk. At ten billion that first touch costs about 11.8 times the warm number on a median query, against 9.8 times at a billion.

Running at this scale turned up something we would not have found otherwise: a handful of our queries blow past the 60 second cap on that first cold pass and then come back in well under a second once the pages are resident. Mostly the BM25 top-k family. That is a good find and we are on it.

Chapter 8. Don't trust us, run SearchBench yourself​

Everything you need is in the repo and it is one shell command.

git clone https://github.com/serenedb/searchbench
cd searchbench/clickhouse # or serenedb
SEARCHBENCH_DATA_DIR=/path/to/data SEARCHBENCH_DATASET=otel_logs_100m ./benchmark.sh --index

Think we tuned something wrong or you would have built a different index? Does a newer release change the picture? Open a PR against the adapter and we will rerun it and republish. Everyone in this field is guessing about everyone else's engine. The only way out of that is to compare notes properly.

Chapter 9. Conclusion​

ClickHouse says work began in 2009 "with the vision of being the fastest OLAP database on earth". We came into this expecting the narrowest gap of the series, because on raw speed they have earned that reputation. On this workload it did not hold.

We answer all 92 queries at a hundred million, at a billion and at ten billion logs, on one box, at 15.0 ms, 31.0 ms and 173.5 ms. ClickHouse cannot express 22 of them at any scale and at ten billion it loses 10 more to the clock. On the queries we both answer we are 2.3x faster at a hundred million, 13.2x at a billion and 18.7x at ten billion. We also load faster and sit in two thirds of the disk. Our table has no sorting key. Theirs does.

Underneath is IResearch, the fastest C++ search engine that won Search Benchmark, The Game, sitting on a columnstore with a vectorized executor above it. The search index and the analytical engine are the same system, which is why the counting and grouping families hold up at ten billion instead of falling back to a scan.

ClickHouse has built real things here. Phrase search with positions is new, it works and that is the hard part. Relevance ranking and the scan-backed shapes are not there yet, but they ship fast and we will be here when they land.

SereneDB puts the whole thing in one system: your indexes, the columnstore and an executor that knows about both. No search cluster sitting next to a warehouse, no glue code keeping two copies of your logs in sync and no queries you have to give up on because the index cannot express them.

The easiest way to disagree with any of this is to run it.

docker run -d --name serenedb -p 7890:7890 serenedb/serenedb

That is the whole install and the quick start takes it from there. Point it at your own logs. If your results look different, open an issue and show us.

If you find this interesting, we'd be grateful if you support SereneDB with a star on GitHub. For an early-stage project, it means more than you might think.

Other SearchBench write-ups:

• SereneDB vs Elasticsearch vs OpenSearch vs CrateDB
• SereneDB vs ParadeDB vs TigerData vs Postgres
• SereneDB vs ArangoDB on search performance
• The C++ search engine that won the game