Skip to main content

Edit this page

split_by_non_alpha

The split_by_non_alpha template cuts the value into maximal runs of ASCII alphanumeric bytes — [0-9A-Za-z] — and emits each run as one token. Every other byte is a separator and is dropped. There is nothing to configure but case conversion: no delimiter list, no pattern, no locale.

That makes it the tokenizer for mixed machine-readable text whose separators are not known in advance — log lines, identifiers, version strings, URLs, serial numbers — where anything that is not a letter or a digit should split. Underscore is a separator here, so 123_abc yields 123 and abc. It is strictly ASCII: every byte from 0x80 up is a separator, so it is not a general-purpose text tokenizer. For Unicode-aware word boundaries use split_text; for a known separator use split_text_csv, and for a separator too complex for a fixed string use split_by_pattern.

As a function: split_by_non_alpha(value, case := 'none') — the value first, then the options in the order below. See tokenizer functions for how a value, a list and a chain of calls behave.

Options​

OptionTypeDefaultDescription
CASEstring'none'Case conversion applied to each token: 'none', 'lower', 'upper'

CASE is the only option this template takes. Its value is matched case-insensitively, so 'Lower' also works; anything outside the three names fails with invalid value in "case" parameter. Any other option — DELIMITER, MIN_GRAM — fails with split_by_non_alpha(): unknown option "<name>". The conversion is ASCII-only, which is exhaustive here, because a token holds nothing but ASCII letters and digits by construction.

Tokenization​

The value is scanned once and cut at every byte that is not an ASCII letter or digit. Tokens are the runs between the cuts, in ascending order, non-overlapping, one token per run. Empty runs are never emitted, so leading, trailing and adjacent separators produce no empty tokens, and an empty value — or a value holding no alphanumeric byte at all — produces no tokens. Positions are implicit and consecutive, one per token in emission order, and offsets are the [start, end) byte range of the run inside the value. Offsets are recorded and are unaffected by case conversion, so all four feature flags — FREQUENCY, POSITION, NORM and OFFSET — are accepted for this template, subject to their dependencies: OFFSET requires POSITION, and POSITION and NORM require FREQUENCY.

The classification is byte-based, not codepoint-based. A letter is A–Z or a–z and a digit is 0–9; _ is a separator, unlike in the word class the other tokenizers use. Every byte from 0x80 up is a separator too, which is what makes the template ASCII-only: a multi-byte UTF-8 letter splits the word around it, so Straße yields Stra and e, and text in a script that has no ASCII letters produces no tokens at all. Bytes that are not valid UTF-8 are not an error either — they are simply separators.

Nothing beyond the split and the optional case conversion happens: accents are not stripped, no Unicode normalization is applied, and there is no stemming and no stop-word removal.

InputOptionsTokens
Hello, World! 123abcdefaults{Hello,World,123abc}
Hello, World! 123_abcdefaults{Hello,World,123,abc}
The Quick-Brown FOX 2024CASE = 'lower'{the,quick,brown,fox,2024}
Straße ÜBER Ab1CASE = 'lower'{stra,e,ber,ab1}

The second row shows the underscore splitting 123_abc in two. The fourth row shows both ASCII-only effects at once: ß and Ü are separators, so Straße becomes stra and e and ÜBER loses its first character, while the ASCII bytes around them are lowercased normally.

Offsets always point at the run in the original value, whatever CASE does to the token text. In the first row Hello covers bytes 0–5 and 123abc covers bytes 14–20. In the fourth row, with CASE = 'lower', stra still covers bytes 0–4, e covers 6–7 and ber covers 10–13 — the two bytes of ß and the two bytes of Ü are counted even though no token contains them.

Examples​

The template needs nothing but its name, so the shortest useful dictionary is one option long:

CREATE TEXT SEARCH DICTIONARY alnum_parts AS
    split_by_non_alpha();

SELECT ts_lexize('alnum_parts', 'Hello, World! 123_abc');

Adding CASE = 'lower' makes matching case-insensitive, and the two feature flags below let the index rank results and answer phrase queries:

CREATE TABLE logs (
    id INTEGER PRIMARY KEY,
    line VARCHAR
);

CREATE TEXT SEARCH DICTIONARY alnum_lower AS
    split_by_non_alpha(case := 'lower')
    WITH (frequency, position);

CREATE INDEX idx_logs ON logs USING inverted (id, line alnum_lower);

The template also composes: it can be a stage of a pipeline (split_by_non_alpha(case := 'lower') | stem_words('en_US.UTF-8')) or a branch of a union.

A token is a maximal run of [A-Za-z0-9]; punctuation, whitespace, underscores and every non-ASCII byte separate, and case := 'lower' folds the ASCII letters. It is the dictionary-free equivalent of regexp_split_to_array(text, '[^A-Za-z0-9]+') without the regex engine:

Query
SELECT split_by_non_alpha('The Quick-Brown FOX 2024', case := 'lower') AS tokens;
Result
 tokens---------------------------- {the,quick,brown,fox,2024}

See also​