Data lake search engine: index Parquet and Iceberg files in place
SereneDB searches Parquet, CSV, JSON and Iceberg data where it sits, on S3 or local disk: a SQL view names the files and an inverted index is built on the view. BM25 full-text search, exact filters and vectors run in SQL, the files stay the source of truth, and REINDEX indexes only what changed.
curl https://install.serenedb.com | shIndex Parquet files in place, and watch what REINDEX changes
Data lake search means searching files where they already sit in object storage, without loading them into another system first. SereneDB's inverted index holds postings for the files as they were at the last pass. Add, overwrite or delete an object and searches keep answering from that snapshot until REINDEX, which indexes only the objects that appeared, changed or disappeared. INCLUDE decides which columns come back without opening a file. The objects below are the walkthrough's; every behaviour shown ran on SereneDB 26.11.2 against an S3 endpoint.
| object | size | in file | in index | status |
|---|---|---|---|---|
| part-000.parquet | 940 B | 1 | 1 | indexed |
| part-001.parquet | 960 B | 1 | 1 | indexed |
| part-002.parquet | 947 B | 2 | 2 | indexed |
Three objects, all indexed. The index holds postings and the INCLUDE columns; the rows stay in the files.
part-003.parquet landed in the bucket. Searches still answer from the last pass; REINDEX indexes just this object.
part-001.parquet was rewritten by its producer: row 4 now reads 'worker 7 ran out of disk in sort'. Until REINDEX the index still matches row 4, and message, read from the file, already shows the new text.
part-000.parquet was deleted. COUNT and the INCLUDE columns still answer from the index; reading message from that object fails until REINDEX drops its postings.
REINDEX indexed the objects that appeared or changed, dropped the deleted ones and left the rest untouched. Readers switched to the new index in one step.
SELECT service, latency_ms, message FROM logs_idx WHERE message @@ ts_phrase('out of memory') AND level @@ 'ERROR';
| service (i) | latency_ms (i) | message (l) |
|---|---|---|
| ingest | 412 | out of memory while indexing batch 17 |
| worker | 2210 | worker 7 ran out of memory in sort |
| ingest | 1830 | out of memory again, retrying batch 19 |
| worker | 2950 | out of memory: sort spilled 4 GB to disk |
The index matches the bucket.
message is read back from the files for the 4 matched rows, in 3 objects. INCLUDE it to answer from the index alone.
A view names the files, the index stores postings and the columns you choose
One view per dataset and one inverted index over it. Text columns get a dictionary for BM25, filter columns exact terms, vector columns IVF, and INCLUDE stores the columns you return. Hover a stage, or a lane inside the index.
Files stay the source of truth
Parquet, CSV and JSON files on local disk or S3, named by one path or one glob. A secret with a SCOPE gives SereneDB read access to a bucket; S3-compatible stores take the same secret with an ENDPOINT, URL_STYLE 'path' and, without TLS, USE_SSL false. A refresh re-reads only the objects that appeared, changed or disappeared.
Iceberg through its REST catalog
CREATE SERVER … iceberg_fdw attaches the catalog, and the view names the table in full. A REINDEX indexes only the snapshot difference, inserts and row-level deletes included, whichever engine committed them. The catalog's own authentication methods replace 'none' outside a local setup.
Hugging Face datasets, read in place
hf:// paths read public datasets on the Hugging Face Hub straight into a view. Our code-search demo indexes about 14,000 programming problems and 11.5 million accepted solutions, about 14 GB of text, read as Parquet from Hugging Face. Below, OpenAI's HumanEval set, 164 problems.
One reader, one glob
A view over one reader and one path or glob is a fast-path source: SereneDB keeps each row's position, re-reads unstored columns by it and refreshes incrementally. A column subset, renames and casts keep the fast path; a UNION ALL or a join makes the view generic, which still indexes and counts but returns only stored columns and rebuilds in full on every refresh. Hive partition columns come from the path.
Postings, not rows
CREATE INDEX … USING inverted on the view reads only the columns it needs from the files. A text column goes through a dictionary for BM25 and phrases, other columns are indexed as exact terms, and INCLUDE stores the columns a query returns, so they come back without opening a file. Field order carries no meaning.
A phrase and an exact match
ts_phrase matches the words in order, and a column without a dictionary matches its exact value. Both run inside one IRESEARCH_SCAN, so they compose with WHERE, JOIN and GROUP BY; ts_highlight marks the match.
BM25 needs frequency and positions
What the index records per token is set on the dictionary: frequency for BM25, position for phrases, offset for highlighting. The same pipeline analyzes the files at build time and every query, so Workers and worker meet as one token.
Find it and measure it in one query
approx_quantile, count and avg over INCLUDE columns run inside the search query. EXPLAIN shows one IRESEARCH_SCAN projecting service and latency_ms, and the S3 endpoint logged no request for it.
INCLUDE columns from the index, the rest from the files
EXPLAIN marks every projected column: (i) is stored in the index, (l) is read from the files for the matched rows only, live. If an object was deleted since the last pass, that read fails until REINDEX; a column stored with INCLUDE keeps answering, as it was at the last pass.
What is stored where
DROP INDEX removes the index; the files stayParquet, CSV, JSON or Iceberg. The source of truth; any engine keeps writing to it.
Derived from the files, so the index can always be rebuilt from them.
New, changed and removed objects; for Iceberg, the snapshot difference. reindex_interval runs it as a job.
A vector column takes an IVF index in the same inverted index, next to BM25.
Make Parquet files in S3 searchable in five statements
Name the files with a view, build one index, and query the lake in SQL from any Postgres client. Nothing is copied, and DROP INDEX leaves the bucket as it was. Tested with SereneDB 26.11.2 on 2026-11-03: every statement and result below ran against an S3-compatible endpoint.
CREATE TEXT SEARCH DICTIONARY english_dict AS split_text(case := 'lower') | stem_words('en_US.UTF-8') WITH (frequency, position, offset); SELECT ts_lexize('english_dict', 'Workers ran out of memory'); -- {worker,ran,out,of,memori}
CREATE SECRET lake_s3 ( TYPE s3, KEY_ID '⟨access key id⟩', SECRET '⟨secret access key⟩', REGION 'us-east-1', SCOPE 's3://my-bucket/' ); CREATE VIEW logs AS SELECT * FROM read_parquet('s3://my-bucket/logs/*.parquet');
CREATE INDEX logs_idx ON logs USING inverted (id, level, service, message english_dict) INCLUDE (service, latency_ms);
SELECT id, service, ts_highlight(message) AS snippet FROM logs_idx WHERE message @@ ts_phrase('out of memory') AND level @@ 'ERROR' ORDER BY BM25(logs_idx.tableoid) DESC, id LIMIT 5;
-- a producer writes s3://my-bucket/logs/part-003.parquet REINDEX INDEX logs_idx; ALTER INDEX logs_idx SET (reindex_interval = 300000); SELECT job_name, schedule FROM sdb_jobs();
What teams build on it
Find the error, then measure it
Application logs land in S3 as Parquet. Full-text search on the message and exact filters on level and service find the lines, and with latency INCLUDEd the p95 comes out of the same query, read from the index.
Real-time analytics →SELECT service, count(*) AS hits, approx_quantile(latency_ms, 0.95) AS p95_ms FROM logs_idx WHERE message @@ ts_phrase('out of memory') GROUP BY service ORDER BY p95_ms DESC;
| service | hits | p95_ms |
|---|---|---|
| worker | 2 | 2950 |
| api | 1 | 2400 |
| ingest | 2 | 1830 |
Hybrid retrieval over a document lake
Chunks and their embeddings sit in Parquet files or an Iceberg table. One index carries BM25 on the text and IVF on the vectors, so a phrase filter and a similarity order run in one statement, and the chunk text is read back for the top rows only.
RAG database →CREATE VIEW chunks AS SELECT tenant_id, source_name, page, body, emb::FLOAT[3] AS emb FROM read_parquet('s3://my-bucket/docs/*.parquet'); CREATE INDEX chunks_idx ON chunks USING inverted ( tenant_id, source_name, body english_dict, emb ivf (metric = 'cosine')); SELECT source_name, page, body FROM chunks_idx WHERE body @@ ts_phrase('scope 3') ORDER BY emb <=> [0.90, 0.12, 0.06]::FLOAT[3] LIMIT 3;
| source_name | page | body |
|---|---|---|
| report.pdf | 1 | carbon emissions grow with scope 3 accounting |
| notes.pdf | 1 | scope 3 emissions of the supply chain |
Contracts by year, searched by clause
History that rarely changes, partitioned by year. The partition column filters inside the index, a text match runs beside it, and filename tells you which object holds each match.
CREATE VIEW contracts AS SELECT * FROM read_parquet('s3://my-bucket/contracts/*/*.parquet', hive_partitioning = true, filename = true); CREATE INDEX contracts_idx ON contracts USING inverted (id, year, party, body english_dict); SELECT id, year, party, filename FROM contracts_idx WHERE body @@ 'termination' AND year = 2025 ORDER BY BM25(contracts_idx.tableoid) DESC, id;
| id | year | party | filename |
|---|---|---|---|
| 201 | 2025 | Acme Corp | s3://my-bucket/contracts/year=2025/data_0.parquet |
| 202 | 2025 | Initech | s3://my-bucket/contracts/year=2025/data_0.parquet |
Data lake search is searching files where they already live in object storage (Parquet, CSV, JSON, Iceberg tables) instead of loading them into a separate search system first. SereneDB reads the files through a SQL view and stores only an inverted index over them, so full-text, filtered and vector queries run against the index while the files stay the source of truth.
SereneDB builds its inverted index directly on a view over the files, on local disk or S3, so the lake is searchable without an ingest pipeline. The same index answers BM25 full-text search, exact filters, aggregates and vector search in one SQL statement. Its published code-search demo indexes about 11.5 million solutions read as Parquet from Hugging Face. SereneDB vs Elasticsearch compares it with an engine that searches the indices you ingest into.
Create a view over the files with read_parquet('s3://…/*.parquet'), then CREATE INDEX … USING inverted on the view. The rows stay in the bucket; the index stores postings and any INCLUDE columns, and REINDEX picks up new files. The five statements above ran as written on SereneDB 26.11.2.
Point read_parquet, read_csv or read_json at a path or a glob, or attach an Iceberg catalog with CREATE SERVER, and query it like a table from any Postgres client: SereneDB speaks the PostgreSQL wire protocol. Add an inverted index on a view when you need ranked text search or fast filtered lookups. For tables in another database, see zero-ETL search.
Yes. Text columns go through a dictionary for BM25 and phrase search, other string columns are indexed as exact terms, and vector columns take IVF. Parquet's own page index holds per-page statistics that let a scan skip pages; an inverted index answers term lookups and ranks the matches.
Yes. Attach the REST catalog with CREATE SERVER … FOREIGN DATA WRAPPER iceberg_fdw, put a view on the table named in full, and index the view. A REINDEX indexes only the snapshot difference, including row-level deletes, whichever engine committed them; see the Iceberg cookbook.
The index keeps serving its last snapshot until REINDEX INDEX, which indexes only the objects that appeared, changed or disappeared and swaps the result in atomically. Columns read from the files in the meantime show their current content, and a read from a deleted object fails until the pass. Set reindex_interval to run the same pass as a job.
A LIKE '%…%' filter reads the text column of every file it cannot skip, on every query, and returns the matches unranked: EXPLAIN shows a READ_PARQUET with a contains() column filter. An inverted index answers @@ from postings, ranks with BM25, and reads the files only for the matched rows' unstored columns.
Yes. Columns listed in INCLUDE are stored in the index, so count, avg or approx_quantile over the matches run inside the search query; the plan shows a single index scan and no file read.
No. The files stay in your bucket; SereneDB stores the postings, any INCLUDE columns and the vector structures it derived from them, so dropping the index loses nothing and it can be rebuilt from the files. Tables in attached external databases can be indexed the same way: see zero-ETL search.
Yes: a view can UNION ALL several readers and map their columns. Such a view is generic, so INCLUDE the columns you return, and expect each refresh to rebuild the index in full; one glob per view keeps refreshes incremental.
Yes. Put a fixed-size vector column in the view (emb::FLOAT[N]) and index it with IVF next to the text column; a BM25 filter and a similarity order then run in one statement. For score fusion such as RRF, see hybrid search.
Parquet, CSV, JSON and NDJSON files, one path or a glob, hive-partitioned or not; Iceberg tables through a REST catalog; and text files as whole documents with read_text. Paths can be local, s3:// on S3 or an S3-compatible store, or hf:// on Hugging Face.
Get started with SereneDB
Point a view at your bucket, build one index, and search the files where they are.