Tokenizes string
and performs folding on each token.
A token is a non-empty sequence of alphanumeric characters in the source string, separated by non-alphanumeric characters. An "alphanumeric" character for this purpose is one that matches [func@GLib.unichar_isalnum] or [func@GLib.unichar_ismark].
Each token is then (Unicode) normalised and case-folded. If ascii_alternates
is non-`NULL` and some of the returned tokens
contain non-ASCII characters, ASCII alternatives will be generated.
The number of ASCII alternatives that are generated and the method for doing so is unspecified, but translit_locale
(if
specified) may improve the transliteration if the language of the source string is known.
ascii_alternates |
a return location for ASCII alternates |
string |
a string to tokenize |
translit_locale |
the language code (like 'de' or 'en_GB') from which |
the folded tokens |