Token interface

Individual token returned by a TokenizerFunction.

Properties

end	End character position of the token within the outer string.
normalized	Normalized form of the token. This can include things like lower casing or stemming.
start	Start character position of the token within the outer string.
text	Original text of the token.

End character position of the token within the outer string.

end: number

number

Normalized form of the token. This can include things like lower casing or stemming.

normalized: string

string

Start character position of the token within the outer string.

start: number

number

Original text of the token.

text: string

string