Secrets Manager

The Secrets manager provides a unified user interface for secrets across all backends that use them. Secrets can be scoped, so different storage prefixes can have different secrets, allowing for example to join data across organizations in a single query. Secrets can also be persisted, so that they do not need to be specified every time SereneDB is launched.

Attention

Persistent secrets are stored in an unencrypted format.

Types of Secrets

Secrets are typed, their type identifies which service they are for. The following secret types are available:

Secret type	Service / protocol
aws	AWS: takes the options of s3 and serves the s3://, r2:// and gcs:// paths that no secret of their own type matches; Iceberg catalogs accept it in place of s3
azure	Azure Blob Storage
clickhouse	ClickHouse
gcs	Google Cloud Storage
http	HTTP and HTTPS
huggingface	Hugging Face
iceberg	Iceberg REST Catalog
openai	OpenAI-compatible model API, used by the AI functions
postgres	PostgreSQL
r2	Cloudflare R2
s3	AWS S3
typesafe	TypeSafe Jev or a local Kev server, used by ai_system_one

sdb_secret_types() also lists rds, which cannot be created.

Per-cloud guides cover how to obtain each kind of credential and which one fits your deployment: Google Cloud credentials (gcs, and iceberg targeting BigLake), AWS credentials (s3, and the S3 Tables / Glue catalogs), and Azure credentials (azure). For iceberg secrets in general — every supported catalog authentication method, and when to use which — see Iceberg catalog authentication. The remaining types are single-credential and documented where they are used: r2, http, huggingface, and postgres; clickhouse is described below. An AI function refuses an openai or typesafe secret whose base_url does not start with https://, unless its host is local (localhost, an address in 127.0.0.0/8 or [::1]) or sdb_ai_allow_insecure_endpoint is on; CREATE SECRET itself accepts such a secret.

For each type, there are one or more "secret providers" that specify how the secret is created. Secrets can also have an optional scope, which is a file path prefix that the secret applies to. When fetching a secret for a path, the secret scopes are compared to the path, returning the matching secret for the path. In the case of multiple matching secrets, the longest prefix is chosen.

Creating a Secret

Secrets can be created using the CREATE SECRET SQL statement. Secrets can be temporary or persistent. Temporary secrets are used by default and live in memory for the lifespan of the SereneDB instance. Persistent secrets are stored in an unencrypted format and are automatically loaded when SereneDB starts.

Secret Providers

To create a secret, a Secret Provider needs to be used. A Secret Provider is a mechanism through which a secret is generated. To illustrate this, the S3 secret type supports the CONFIG provider, which takes all configuration information in the CREATE SECRET statement. When no Secret Provider is specified, the CONFIG provider is used. The S3 credential_chain provider, which fetches credentials automatically, needs DuckDB's aws extension and is not available in SereneDB; the Azure credential_chain provider is. See AWS Credentials for the ways to authenticate to S3.

Temporary Secrets

To create a temporary unscoped secret to access S3, we can now use the following:

SQL logic test

CREATE SECRET my_secret (    TYPE s3,    KEY_ID 'my_secret_key',    SECRET 'my_secret_value',    REGION 'my_region');

Note that we implicitly use the default CONFIG secret provider here.

Persistent Secrets

In order to persist secrets between SereneDB database instances, we can now use the CREATE PERSISTENT SECRET command, e.g.:

SQL logic test

CREATE PERSISTENT SECRET my_persistent_secret (    TYPE s3,    KEY_ID 'my_secret_key',    SECRET 'my_secret_value');

This writes the secret (unencrypted) so that it is available again the next time SereneDB starts.

Deleting Secrets

Secrets can be deleted using the DROP SECRET statement, e.g.:

SQL logic test

DROP PERSISTENT SECRET my_persistent_secret;

Creating Multiple Secrets for the Same Service Type

If two secrets exist for a service type, the scope can be used to decide which one should be used. For example:

SQL logic test

CREATE SECRET secret1 (    TYPE s3,    KEY_ID 'my_secret_key1',    SECRET 'my_secret_value1',    SCOPE 's3://⟨my-bucket⟩');

SQL logic test

CREATE SECRET secret2 (    TYPE s3,    KEY_ID 'my_secret_key2',    SECRET 'my_secret_value2',    SCOPE 's3://⟨my-other-bucket⟩');

Now, if the user queries something from s3://⟨my-other-bucket⟩/something, secret secret2 will be chosen automatically for that request. To see which secret is being used, the which_secret table function can be used, which takes a path and a secret type as parameters:

SQL logic test

FROM which_secret('s3://⟨my-other-bucket⟩/file.parquet', 's3');

ClickHouse Secrets

A clickhouse secret holds the connection options of a ClickHouse server: host, port, user, password, database, secure and compression (none, zstd or lz4), or a uri that gives them all. ATTACH '' AS name (TYPE clickhouse, SECRET secret_name) connects with a named secret. Without SECRET, a ClickHouse ATTACH, and a clickhouse_fdw server, take every option their connection string leaves out from the unnamed clickhouse secret (__default_clickhouse), if there is one; a connection string in URI form ignores it.

sql

CREATE SECRET (TYPE clickhouse, host 'clickhouse.internal', user 'reader', password 'secret', compression 'zstd');
ATTACH 'database=events' AS analytics (TYPE clickhouse);

Listing Secrets

Secrets can be listed using the built-in table-producing function, e.g., by using the sdb_secrets() table function:

SQL logic test

FROM sdb_secrets();

sdb_secrets() shows some sensitive values as redacted, but it does not hide every credential a secret holds, so do not rely on it to keep credentials from the users who can call it. For an s3, r2, gcs or aws secret, secret, session_token, http_proxy_password and refresh_info are redacted.