Add custom analyzers to string fields in an Azure AI Search index

A custom analyzer is a user-defined combination of one tokenizer, one or more token filters, and one or more character filters. A custom analyzer is specified within a search index, and then referenced by name on field definitions that require custom analysis. A custom analyzer is invoked on a per-field basis. Attributes on the field will determine whether it's used for indexing, queries, or both.

In a custom analyzer, character filters prepare the input text before it's processed by the tokenizer (for example, removing markup). Next, the tokenizer breaks text into tokens. Finally, token filters modify the tokens emitted by the tokenizer. For concepts and examples, see Analyzers in Azure AI Search.

Why use a custom analyzer?

A custom analyzer gives you control over the process of converting text into indexable and searchable tokens by allowing you to choose which types of analysis or filtering to invoke, and the order in which they occur.

Create and assign a custom analyzer if none of default (Standard Lucence), built-in, or language analyzers are sufficient for your needs. You might also create a custom analyzer if you want to use a built-in analyzer with custom options. For example, if you wanted to change the maxTokenLength on Standard, you would create a custom analyzer, with a user-defined name, to set that option.

Scenarios where custom analyzers can be helpful include:

• Using character filters to remove HTML markup before text inputs are tokenized, or replace certain characters or symbols.

• Phonetic search. Add a phonetic filter to enable searching based on how a word sounds, not how it’s spelled.

• Disable lexical analysis. Use the Keyword analyzer to create searchable fields that aren't analyzed.

• Fast prefix/suffix search. Add the Edge N-gram token filter to index prefixes of words to enable fast prefix matching. Combine it with the Reverse token filter to do suffix matching.

• Custom tokenization. For example, use the Whitespace tokenizer to break sentences into tokens using whitespace as a delimiter

• ASCII folding. Add the Standard ASCII folding filter to normalize diacritics like ö or ê in search terms.

Create a custom analyzer

To create a custom analyzer, specify it in the analyzers section of an index at design time, and then reference it on searchable, Edm.String fields using either the analyzer property, or the indexAnalyzer and searchAnalyzer pair.

An analyzer definition includes a name, type, one or more character filters, a maximum of one tokenizer, and one or more token filters for post-tokenization processing. Character filters are applied before tokenization. Token filters and character filters are applied from left to right.

• Names in a custom analyzer must be unique and can't be the same as any of the built-in analyzers, tokenizers, token filters, or characters filters. Names consist of letters, digits, spaces, dashes or underscores. Names must start and end with plain text characters. Names must be under 128 characters in length.

• Type must be #Microsoft.Azure.Search.CustomAnalyzer.

• charFilters can be one or more filters from Character Filters, processed before tokenization, in the order provided. Some character filters have options, which can be set in a charFilters section. Character filters are optional.

• tokenizer is exactly one Tokenizer. A value is required. If you need more than one tokenizer, you can create multiple custom analyzers and assign them on a field-by-field basis in your index schema.

• tokenFilters can be one or more filters from Token Filters, processed after tokenization, in the order provided. For token filters that have options, add a tokenFilter section to specify the configuration. Token filters are optional.

Analyzers must not produce tokens longer than 300 characters, or indexing will fail. To trim long token or to exclude them, use the TruncateTokenFilter and the LengthTokenFilter respectively. See Token filters for reference.

"analyzers":(optional)[
   {
      "name":"name of analyzer",
      "@odata.type":"#Microsoft.Azure.Search.CustomAnalyzer",
      "charFilters":[
         "char_filter_name_1",
         "char_filter_name_2"
      ],
      "tokenizer":"tokenizer_name",
      "tokenFilters":[
         "token_filter_name_1",
         "token_filter_name_2"
      ]
   },
   {
      "name":"name of analyzer",
      "@odata.type":"#analyzer_type",
      "option1":value1,
      "option2":value2,
      ...
   }
],
"charFilters":(optional)[
   {
      "name":"char_filter_name",
      "@odata.type":"#char_filter_type",
      "option1":value1,
      "option2":value2,
      ...
   }
],
"tokenizers":(optional)[
   {
      "name":"tokenizer_name",
      "@odata.type":"#tokenizer_type",
      "option1":value1,
      "option2":value2,
      ...
   }
],
"tokenFilters":(optional)[
   {
      "name":"token_filter_name",
      "@odata.type":"#token_filter_type",
      "option1":value1,
      "option2":value2,
      ...
   }
]

Within an index definition, you can place this section anywhere in the body of a create index request but usually it goes at the end:

{
  "name": "name_of_index",
  "fields": [ ],
  "suggesters": [ ],
  "scoringProfiles": [ ],
  "defaultScoringProfile": (optional) "...",
  "corsOptions": (optional) { },
  "analyzers":(optional)[ ],
  "charFilters":(optional)[ ],
  "tokenizers":(optional)[ ],
  "tokenFilters":(optional)[ ]
}

The analyzer definition is a part of the larger index. Definitions for char filters, tokenizers, and token filters are added to the index only if you're setting custom options. To use an existing filter or tokenizer as-is, specify it by name in the analyzer definition. For more information, see Create Index (REST). For more examples, see Add analyzers in Azure AI Search.

Test custom analyzers

You can use the Test Analyzer (REST) to see how an analyzer breaks given text into tokens.

Request

  POST https://[search service name].search.windows.net/indexes/[index name]/analyze?api-version=[api-version]
    Content-Type: application/json
    api-key: [admin key]

  {
     "analyzer":"my_analyzer",
     "text": "Vis-à-vis means Opposite"
  }

Response

  {
    "tokens": [
      {
        "token": "vis_a_vis",
        "startOffset": 0,
        "endOffset": 9,
        "position": 0
      },
      {
        "token": "vis_à_vis",
        "startOffset": 0,
        "endOffset": 9,
        "position": 0
      },
      {
        "token": "means",
        "startOffset": 10,
        "endOffset": 15,
        "position": 1
      },
      {
        "token": "opposite",
        "startOffset": 16,
        "endOffset": 24,
        "position": 2
      }
    ]
  }

Update custom analyzers

Once an analyzer, a tokenizer, a token filter, or a character filter is defined, it can't be modified. New ones can be added to an existing index only if the allowIndexDowntime flag is set to true in the index update request:

PUT https://[search service name].search.windows.net/indexes/[index name]?api-version=[api-version]&allowIndexDowntime=true

This operation takes your index offline for at least a few seconds, causing your indexing and query requests to fail. Performance and write availability of the index can be impaired for several minutes after the index is updated, or longer for very large indexes, but these effects are temporary and eventually resolve on their own.

Built-in analyzers

If you want to use a built-in analyzer with custom options, creating a custom analyzer is the mechanism by which you specify those options. In contrast, to use a built-in analyzer as-is, you simply need to reference it by name in the field definition.

¹ Analyzer Types are always prefixed in code with #Microsoft.Azure.Search such that PatternAnalyzer would actually be specified as #Microsoft.Azure.Search.PatternAnalyzer. We removed the prefix for brevity, but the prefix is required in your code.

The analyzer_type is only provided for analyzers that can be customized. If there are no options, as is the case with the keyword analyzer, there's no associated #Microsoft.Azure.Search type.

Character filters

Character filters add processing before a string reaches the tokenizer.

Azure AI Search supports character filters in the following list. More information about each one can be found in the Lucene API reference.

¹ Char Filter Types are always prefixed in code with #Microsoft.Azure.Search such that MappingCharFilter would actually be specified as #Microsoft.Azure.Search.MappingCharFilter. We removed the prefix to reduce the width of the table, but please remember to include it in your code. Notice that char_filter_type is only provided for filters that can be customized. If there are no options, as is the case with html_strip, there's no associated #Microsoft.Azure.Search type.

Tokenizers

A tokenizer divides continuous text into a sequence of tokens, such as breaking a sentence into words, or a word into root forms.

Azure AI Search supports tokenizers in the following list. More information about each one can be found in the Lucene API reference.

¹ Tokenizer Types are always prefixed in code with #Microsoft.Azure.Search such that ClassicTokenizer would actually be specified as #Microsoft.Azure.Search.ClassicTokenizer. We removed the prefix to reduce the width of the table, but please remember to include it in your code. Notice that tokenizer_type is only provided for tokenizers that can be customized. If there are no options, as is the case with the letter tokenizer, there's no associated #Microsoft.Azure.Search type.

Token filters

A token filter is used to filter out or modify the tokens generated by a tokenizer. For example, you can specify a lowercase filter that converts all characters to lowercase. You can have multiple token filters in a custom analyzer. Token filters run in the order in which they're listed.

In the table below, the token filters that are implemented using Apache Lucene are linked to the Lucene API documentation.

¹ Token Filter Types are always prefixed in code with #Microsoft.Azure.Search such that ArabicNormalizationTokenFilter would actually be specified as #Microsoft.Azure.Search.ArabicNormalizationTokenFilter. We removed the prefix to reduce the width of the table, but please remember to include it in your code.

See also

• Azure AI Search REST APIs
• Analyzers in Azure AI Search (Examples)
• Create Index (REST)