GPT3Tokenizer class

Reference

Package:: @microsoft/teams-ai

Tokenizer that uses GPT-3's encoder.

Methods

decode(number[])
encode(string)

Method Details

decode(number[])

function decode(tokens: number[]): string

Parameters

tokens: number[]

Returns

string

encode(string)

function encode(text: string): number[]

Parameters

text: string

Returns

number[]