AddedToken Struct
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Represent a token added by the user on top of the existing Model vocabulary. AddedToken can be configured to specify the behavior they should have in various situations like:
- Whether they should only match single words
- Whether to include any WhiteSpace on its left or right
public struct AddedToken : IEquatable<Microsoft.ML.Tokenizers.AddedToken>type AddedToken = structPublic Structure AddedToken
Implements IEquatable(Of AddedToken)- Inheritance
-
AddedToken
- Implements
Constructors
| AddedToken() |
Create a new AddedToken object. |
| AddedToken(String, Boolean) |
Create a new AddedToken object from the given content, specifying if it is intended to be a special token. Special tokens are not normalized by default. |
Properties
| Content |
Gets or sets the content of the added token |
Methods
| Equals(AddedToken) |
Determines whether two token instances are equal. |
| GetHashCode() |
Returns the hash code for the current token. |
Operators
| Implicit(String to AddedToken) |
Defines an implicit conversion of a string object to AddedToken. |