Microsoft.ML.Tokenizers Namespace

Represent the Byte Pair Encoding model.

Allows decoding Original BPE by joining all the tokens and then replacing the suffix used to identify end-of-words by white spaces

The Bpe trainer responsible to train the Bpe model.

Represent the Byte Pair Encoding model.

Normalize the string to lowercase form before processing it with the tokenizer.

Represents a model used during Tokenization (like BPE or Word Piece or Unigram).

Normalize the string before processing it with the tokenizer.

Base class for all pre-tokenizers classes. The PreTokenizer is in charge of doing the pre-segmentation step.

The pre-tokenizer for Roberta English tokenizer.

This Split contains the underlying split token as well as its offsets in the original string. These offsets are in the original referential. It also contains any Token associated to the current split.

Represent the token produced from the tokenization process containing the token substring, the id associated to the token substring, and the offset mapping to the original string.

A Tokenizer works as a pipeline. It processes some raw text as input and outputs a TokenizerResult object.

A Decoder has the responsibility to merge the given list of tokens in a string.

The Encoding represents the output of a Tokenizer.

A Trainer has the responsibility to train a model. We feed it with lines/sentences and then it can train the given Model.

Normalize the string to uppercase form before processing it with the tokenizer.

The pre-tokenizer which split the text at the word boundary. The word is a set of alphabet, numeric, and underscore characters.

Represent a token added by the user on top of the existing Model vocabulary. AddedToken can be configured to specify the behavior they should have in various situations like:

• Whether they should only match single words
• Whether to include any WhiteSpace on its left or right

Contains the normalized string and the mapping to the original string.

Represent the state of the reported progress.