TFDS now supports the Croissant 🥐 format! Read the documentation to know more.

tfds.deprecated.text.Tokenizer

Splits a string into tokens, and joins them back.

tfds.deprecated.text.Tokenizer(
    alphanum_only=True, reserved_tokens=None
)

Args
`alphanum_only`	`bool`, if `True`, only parse out alphanumeric tokens (non-alphanumeric characters are dropped); otherwise, keep all characters (individual tokens will still be either all alphanumeric or all non-alphanumeric).
`reserved_tokens`	`list<str>`, a list of strings that, if any are in `s`, will be preserved as whole tokens, even if they contain mixed alphanumeric/non-alphanumeric characters.

Attributes
`alphanum_only`
`reserved_tokens`

Attributes

alphanum_only

reserved_tokens

join(
    tokens
)

Joins tokens into a string.

@classmethod
load_from_file(
    filename_prefix
)

save_to_file(
    filename_prefix
)

tokenize(
    s
)

Splits a string into tokens.