AddedToken Constructors
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Overloads
| AddedToken() |
Create a new AddedToken object. |
| AddedToken(String, Boolean) |
Create a new AddedToken object from the given content, specifying if it is intended to be a special token. Special tokens are not normalized by default. |
AddedToken()
Create a new AddedToken object.
public AddedToken ();Public Sub New ()Applies to
AddedToken(String, Boolean)
Create a new AddedToken object from the given content, specifying if it is intended to be a special token. Special tokens are not normalized by default.
public AddedToken (string content, bool special = false);new Microsoft.ML.Tokenizers.AddedToken : string * bool -> Microsoft.ML.Tokenizers.AddedTokenPublic Sub New (content As String, Optional special As Boolean = false)Parameters
- content
- String
The content of the added token.
- special
- Boolean
Indicate whether this token is special.