Search Over Your Data Lake — Data Lake Query Without the Full Scan
The index is built over the files where they already sit — Parquet in S3 or on local disk — and nothing is copied into the database.
Search and analytics come out of the same SQL statement, so there is no second system holding a second copy of the lake.
What Is a Data Lake Search?
Your data already lives in object storage in open formats — Parquet, CSV, ORC. That is cheap to keep and easy to read, and it has no search. The usual answer is to copy it somewhere that does: a search cluster or a warehouse, with a pipeline in between. Now you own two copies and the job that keeps them in step.
A data lake search engine does the opposite: it searches the files where they are. In SereneDB a view makes the remote files look like a table, and an inverted index is built on that view — data lake indexing that stores postings, not rows. The corpus stays in the bucket; the index is the only thing the database keeps, and a query reads it instead of scanning the files.
s3://bucket/logs/2025/*.parquet
│
CREATE VIEW logs
│
CREATE INDEX logs_idx
│
WHERE message @@ 'oom'Why Searching a Data Lake Is Hard
Two problems make data lake indexing painful, and both come from the same decision: moving the data somewhere else first.
The copy-and-sync problem
To run a data lake query over data in S3 you normally copy it into Elasticsearch, Solr or a warehouse. That is an ETL pipeline to write, schedule, monitor and pay for — and the moment the files change, the copy is stale. The freshness of your search becomes a property of a cron job.
The schema fragmentation problem
Files arrive from different producers with different schemas. Folding them into one searchable index the traditional way means a mapping per source, kept in sync with every producer that changes a field name. The mapping becomes its own maintenance surface.
How indexing in place solves both
SereneDB builds the index over a view, and the view reconciles the sources with UNION ALL and casts — schema mapping as SQL you can read and diff. The index is the only thing stored; when the files change a refresh pass updates it. No copy of the data, no sync pipeline.
How SereneDB Runs a Data Lake Query
Three statements and one idea: a view names the files, an index covers the columns you search, and INCLUDE decides what comes back without touching the bucket. That is the whole data lake search engine.
Views over remote files
read_parquet makes remote files addressable as a SQL table — a glob across thousands of objects is one name. Several sources with different shapes become one table through UNION ALL and casts, which is where parquet indexing across producers stops being a mapping file.
CREATE VIEW logs AS
SELECT * FROM read_parquet('s3://my-bucket/logs/2025/*.parquet');Inverted index on a view
The index is built on the view, not on a table: text columns get a dictionary for BM25 ranking, structured columns are indexed for exact matching. It is a static snapshot of the files as of build time — which is the honest trade for never copying them.
CREATE INDEX logs_idx ON logs
USING inverted (id, level, service, message english_dict);INCLUDE — columns that ride in the index
INCLUDE(...) stores extra columns beside the posting lists, so a query returns whole rows without reading the source files. Aggregates over those columns — approx_quantile, count, avg — run in the same statement as the search.
CREATE INDEX solutions_idx ON solutions_v
USING inverted (id, task_id, lang, code code_grams)
INCLUDE (id, task_id, code, code_len, lang, time_ms, memory_kb);Step-by-Step: Index and Search Parquet on S3 in 5 Minutes
Create the dictionary
The text search dictionary decides tokenization. position and offset are what make phrase queries and highlighting possible.
CREATE TEXT SEARCH DICTIONARY english_dict (
template = 'text',
locale = 'en_US.UTF-8',
case = 'lower',
stemming = true,
accent = false,
frequency = true,
position = true,
offset = true
);Create a view over the files on S3
The glob resolves at query time; nothing is read yet. Add more sources with UNION ALL when their schemas differ.
CREATE VIEW logs AS
SELECT * FROM read_parquet('s3://my-bucket/logs/2025/*.parquet');Build the inverted index
This statement is the only point where the files are read end to end. What it leaves behind is the index — not a copy of the lake.
CREATE INDEX logs_idx ON logs
USING inverted (id, level, service, message english_dict);Full-text search with a filter
A phrase match on the message, an exact match on the level, ranked by BM25 — one statement, served from the index.
SELECT id, level, message
FROM logs_idx
WHERE message @@ ts_phrase('out of memory') AND level @@ 'ERROR'
ORDER BY BM25(logs_idx.tableoid) DESC, id;Refresh when the files change
The index is a snapshot of the files. REINDEX runs one refresh pass: over a glob it indexes only the files that appeared, changed or disappeared, and readers see the old index or the new one, never half of each. A reindex_interval runs the same pass on a loop.
REINDEX INDEX logs_idx;Use Cases for Data Lake Search
Log analysis at scale
Terabytes of Parquet logs in S3. Full-text search on the message, exact filters on severity and service, BM25 ranking when you do not know which line matters yet. With latency columns INCLUDEd, p50 and p95 come out of the same data lake query as the search — no export step between "find it" and "measure it".
Code search across repositories
Our own code search blog post runs on this shape: 14 000 tasks and 11.5 million solutions, about 14 GB of text, as Parquet on Hugging Face. One view unions several datasets; runtime and memory statistics are returned beside the matching code because they ride in the index.
Archival and compliance data lake queries
History that rarely changes but has to stay searchable: financial transactions, medical records, legal documents in Parquet. A static snapshot is the right shape for data that does not move: parquet indexing runs once, and the index holds a small fraction of the size of the corpus it searches.
Frequently Asked Questions
Searching the data where it already lives in object storage, rather than copying it into a separate search system first. A data lake search engine reads the open files in place and keeps only the index it built over them.
Data lake indexing is two statements: CREATE VIEW over read_parquet or read_csv, then CREATE INDEX … USING inverted on that view. The rows are never copied — the index is the only thing stored.
Yes — parquet indexing works straight off read_parquet('s3://…'), including globs across many objects. CSV, ORC, local files and Hugging Face datasets are supported the same way.
No. The view reads the files in place and the index is built over the view, so a data lake query touches postings rather than a second copy. Losing the index means rebuilding it, not restoring your data. A table in PostgreSQL or ClickHouse can be indexed in place the same way: see zero-ETL search.
Yes. INCLUDEd columns are stored in the index, so approx_quantile, count and avg run inside the search query without going back to the source files.
Run REINDEX INDEX. The index is a snapshot of the files, and a refresh pass over a glob indexes only the files that appeared, changed or disappeared; set reindex_interval on the index to run the pass on a loop.
Yes — the same inverted index takes text columns for BM25 and vector columns for IVF ANN, so hybrid search over the lake is one SQL query, via Filtered ANN or RRF.
Get Started with SereneDB
Point a view at your bucket, build one index, and the lake answers queries. DROP INDEX reverses the experiment.