Transparency note: Azure AI Search

AI enrichment is the application of machine learning models from Azure AI services over content that is not easily searchable in its raw form. Through enrichment, analysis and inference are used to create searchable content and structure where none previously existed.

AI enrichment is an optional extension of the Azure AI Search indexer pipeline that connects to Azure AI services in the same region as a customer's search service. An enrichment pipeline has the same core components as a typical indexer (indexer, data source, index), plus a skill set that specifies the atomic enrichment steps. A skill set can be assembled by using built-in skills based on the Azure AI services APIs, such as Azure AI Vision and Azure AI Language, or custom skills that run external code that you provide.

Vector search is a method of information retrieval where documents and queries are represented in an index as vectors instead of plain text. In vector search, machine learning models, hosted externally from Azure AI Search, generate the vector representations of source inputs, which can be text, images, audio, or video content. This mathematical and normalized representation of content, called vector embeddings, provides a common basis for search scenarios.

When everything is a vector, a query can find a match in vector space, even if the associated original content is in a different media type, such as images versus text, or language than the query. The search engine scans the index looking for vector content that is most similar, that is, the closest, to the vector in the query. Matching on a mathematical vector representation instead of keywords makes it far more likely to find matches that share semantic meaning but are textually distinct, such as "car" and "auto," for example. This gives a more detailed introduction to vector embeddings and how the similarity algorithm works.

Key terms

Semantic ranker uses the context or semantic meaning of a query to compute a new relevance score that promotes results that are semantically closest to the intent of the original query to the top. The initial result set can come from a keyword search with BM25 ranking, vector search, or a hybrid search that includes both. It also creates and returns "captions" by extracting verbatim content found in the result and "highlights" to call attention to important content within the result. It can also return an "answer" if the query has the characteristics of a question ("what is the freezing point of water") and the result contains text having the characteristics of an answer ("water freezes at 0°C or 32°F").

Key terms

System behavior

Several built-in skills for AI enrichment in Azure AI Search take advantage of Azure AI services. See the Transparency Notes for each built-in skill linked below for considerations when choosing to use a skill:

• Key Phrase Extraction Skill: Azure AI Language - Key Phrase Extraction
• Language Detection Skill: Azure AI Language - Language Detection
• Entity Linking Skill: Azure AI Language - Entity Linking
• Entity Recognition Skill: Azure AI Language - Named Entity Recognition (NER)
• PII Detection Skill: Azure AI Language - PII Detection
• Sentiment Skill: Azure AI Language - Sentiment Analysis
• Image Analysis Skill: Azure AI Vision - Image Analysis
• OCR Skill: Azure AI Vision - OCR

Please see the documentation for each skill to learn more about their respective capabilities, limitations, performance, evaluations, and methods for integration and responsible use. Note that using these skills in combination may lead to compounding effects (for example, errors introduced when using OCR will carry through when using key phrase extraction).

Use cases

Example use cases

Because Azure AI Search is a full text search solution, the purpose of AI enrichment is to improve the search utility of unstructured content. Here are some examples of content enrichment scenarios supported by the built-in skills:

• Translation and language detection enable multilingual search.
• Entity recognition extracts people , places , and other entities from large chunks of text.
• Key phrase extraction identifies and then outputs important terms.
• OCR recognizes printed and handwritten text in binary files.
• Image analysis describes image content and outputs the descriptions as searchable text fields.
• Integrated vectorization is a preview feature that calls the Azure OpenAI embeddings model to vectorize data and store embeddings in Azure AI Search for similarity search.

System behavior

In vector search, the search engine looks for vectors within the embedding space in the index to find those close to the query vector. This technique is called nearest neighbor search. This also helps quantify the degree of similarity, or distance, between items. A high degree of vector similarity indicates that the original data was similar too. The two vector search algorithms supported by Azure AI Search have different approaches to this problem, trading off different characteristics such as latency, throughput, recall, and memory.

Finding the true set of “k” nearest neighbors requires comparing the input vector exhaustively against all vectors in the dataset. While each vector similarity calculation is relatively fast, performing these exhaustive comparisons across large datasets is computationally expensive and slow because of the sheer number of comparisons that are required. Also, the higher the dimensionality of each vector, the more complex and slower the calculations on each vector will be.

To address this challenge, approximate nearest neighbor (ANN) search methods are used to trade off recall for speed. These methods can efficiently find a small set of candidate vectors that are most likely to be similar to the query vector, reducing the total number of vector comparisons. Azure AI Search uses the Hierarchical Navigable Small World (HNSW) algorithm to organize high-dimensional data points into a probabilistic hierarchical graph structure that enables fast similarity search while allowing a tunable trade-off between search accuracy and computational cost.

Azure AI Search also supports multiple similarity metrics to determine nearest neighbor and score of each vector result These include cosine, “Euclidean” (also known as “l2 norm”), and “dot product.” Cosine calculates the angle between two vectors. Euclidean calculates the Euclidean distance between two vectors, which is the l2-norm of the difference of the two vectors. Dot product is affected by both vectors' magnitudes and the angle between them. For normalized embedding spaces, dot product is equivalent to the cosine similarity but is more efficient.

Use cases

Example use cases

There are many scenarios where vector search is useful, and they are limited only by the capabilities of the model used to generate vector embeddings. Here are some general use cases where vector search can be used:

• Semantic search : Extract semantic understanding from text by using a model, like using models such as the Azure OpenAI Service embeddings models.
• Search across different data types (multimodal): Encode content coming from images, text, audio, and video, or a mix, and do a single search across all of them.
• Multilingual search :Use a multilingual embeddings model to represent your document in multiple languages to find results in supported languages.
• Hybrid search :Vector search is implemented at the field level, which means you can build queries that include vector fields and searchable text fields. The queries run in parallel, and the results are merged into a single response. Hybrid search results with semantic ranking have been shown to provide the best qualitative results.
• Filtered vector search : A query can include a vector query and filter expression. Filters applied to other data types are useful for including or excluding documents based on other criteria.
• Vector database :This pure vector store is for long-term memory or an external knowledge base for Large Language Models (LLMs). For example, use Azure AI Search as a vector index in Azure Machine Learning prompt flow for Retrieval Augmented Generation (RAG) applications.

Considerations when choosing a use case

There may be considerations and concerns associated with the specific model you choose to generate vector embeddings. Each model could have its own issues with bias and fairness and should be evaluated before being used in your application. Azure AI Search does not provide any models to vectorize content as part of the service. See the Azure OpenAI Service Transparency Note for examples of these considerations. Other third-party or OSS models will have considerations of their own to review.

System behavior

Ranking the results of the first layer retrieval step is a highly resource-intensive process. To complete the ranker’s processing within the expected latency of a query operation, only the top 50 results from the retrieval engine are sent to the semantic ranker as inputs. If too long, the 50 results are first sent to a summarization step that extracts the most relevant content from each result before running the semantic ranker.

In the summarization step, the retrieved document is first put through a preparation process that concatenates the different document inputs into a single long string. If the string is too long, a trimming exercise takes place, with particular emphasis placed on retaining content contained within fields added to the semantic configuration. After strings are prepared, they are passed through machine reading comprehension and language representation models to determine which sentences and phrases provide the best summary, relative to the query. This phase extracts content from the string that will be passed forward to the semantic ranking stage and optionally outputs a semantic caption or semantic answer.

The final step, semantic ranking, determines the relevance of the content extracted in the prior step to the user’s query and outputs a semantic ranking score ranging from 4 (highly relevant) to 0 (irrelevant). This step is based on the query text and on the summarized text and involves more complex computations than those of the retrieval layer.

Use cases

Example use cases

Semantic ranker can be used in multiple scenarios. The system's intended use cases include:

• Retrieval Augmented Generation (RAG): Semantic ranker enables you to ground responses from your generative AI applications in relevant search results that meet the relevancy score threshold that you define. For example, the Azure OpenAI Service on your data uses Azure AI Search to augment Azure OpenAI models with your data. You can use semantic ranker within this service to improve the relevancy of the information fed to the Azure OpenAI model.
• Content search: Semantic ranker allows you to search for relevant content within your data by analyzing text and metadata. For example, search on the learn.microsoft.com website uses semantic ranker to improve the search relevance for software developers who search for Microsoft technical documentation.
• eCommerce search: Semantic ranker enables eCommerce businesses to enhance their search experience by providing relevant product results based on semantic relevance. For example, online retailers use semantic ranker to optimize their eCommerce experience by providing relevant search results for their online shoppers.
• QnA: Azure AI Search enables organizations to provide a conversational experience for their users by answering questions based on the information available in their databases. For example, a manufacturer can use semantic ranker to augment information available to a chatbot. Engineers can use this chatbot to ask questions and retrieve highly relevant internal documents related to their query and instant answers within the retrieved documents.

Considerations when choosing a use case

We encourage customers to use semantic ranker in their innovative solutions or applications. However, here are some considerations when choosing a use case:

• Sensitive information : The machine learning models that enable semantic ranker process the data retrieved in a search query, including sensitive information such as personal details and financial information. Consider any privacy and security implications before you implement semantic ranker for such use cases.
• Bias and fairness : Semantic ranker is powered by deep learning models. These deep learning models were trained by using public content. Customer data is scored by the semantic ranker models. Evaluate the output of semantic ranker when you select use cases, especially for use cases that have implications for fairness and equity, such as hiring and recruitment.
• Regulation compliance: Some industries, such as healthcare and finance, are highly regulated and may have restrictions on the use of AI and machine learning. Before you use semantic ranker in such industries, ensure that the solution complies with relevant regulations and guidelines.

AI enrichment in Azure AI Search uses the indexer and data source features of the service to call Azure AI services to perform the content enrichment. Limitations of the indexers and data sources used in this process will apply. Review the indexer and data source documentation for more information about these related limitations. The limitations of each Azure AI Service used by the AI enrichment pipeline in Azure AI Search will also apply. See the transparency notes for each service for more information about these limitations.

Technical limitations, operational factors, and ranges

All vectors uploaded to Azure AI Search must be generated externally from the service by using a model of your choice. It is your responsibility to consider the technical limitations and operating factors of each model, and whether the embeddings it creates are well optimized or even appropriate for your use case. This includes both the inferences of meaning extracted from content, and the dimensionality of the vector embedding space.

The vectorization model creates an embeddings space that defines the resulting end user search experience of an application. There could be downsides to a model that negatively affects both functionality and performance if a model does not align well with a desired use case or the embeddings generated are poorly optimized.

While many limitations of vector search will stem from the model used to generate embeddings, there are some additional options you should consider at query time. You can choose from two algorithms to determine relevance for vector search results: Exhaustive k-nearest neighbors (KNN) or Hierarchical Navigable Small World. Exhaustive k-nearest neighbors (KNN) performs a brute-force search of the entire vector space for matches that are most similar to the query by calculating the distances between all pairs of data points and finding the exact k nearest neighbors of a query point. While more precise, this algorithm can be slow. If low latency is the primary goal, consider using the Hierarchical Navigable Small World (HNSW) algorithm. HNSW performs an efficient approximate nearest neighbor (ANN) search in high-dimensional embedding spaces. See the vector search documentation for more information about these options.

Best practices for improving system performance

• Do spend time A/B testing your application with the different content and query types you expect your application to support. Figure out which query experience is best for your needs.
• Do spend time testing your models with a full range of input content to understand how it behaves in many situations. This content could include potentially sensitive input to understand whether there is any bias inherent in the model. The Azure OpenAI Responsible AI overview provides guidance for how to responsibly use AI.
• Do consider adding Azure AI Content Safety to your application architecture. It includes an API to detect harmful user-generated and AI-generated text or images in applications and services.

Evaluating and integrating vector search for your use

To ensure optimal performance, conduct your own evaluations of the solutions you plan to implement by using vector search. Follow an evaluation process that: (1) uses some internal stakeholders to evaluate results, (2) uses A/B experimentation to roll out vector search to users, (3) incorporates key performance indicators (KPIs) and metrics monitoring when the service is deployed in experiences for the first time, and (4) tests and tweaks the semantic ranker configuration and/or index definition, including the surrounding experiences like user interface placement or business processes.

Microsoft has rigorously evaluated vector search both in terms of latency and recall and relevance by leveraging diverse datasets to measure the speed, scalability, and accuracy of results returned. The primary focus of your evaluation efforts should be on selecting the appropriate model for your specific use case, understanding the limitations and biases of the model, and rigorously testing the end to end vector search experience.

Technical limitations, operational factors, and ranges

There may be cases where semantic results, captions, and answers may not appear to be correct. The models used by semantic ranker are trained on various data sources (including open source and selections from the Microsoft Bing corpus). Semantic ranker supports a broad range of languages and attempts to match user queries to content from your search results. Semantic ranker is also a premium feature at additional cost which should be considered when projecting the overall cost of your end-to-end solution.

Semantic ranker is most likely to improve relevance over content that is semantically rich, such as articles and descriptions. It looks for context and relatedness among terms, elevating matches that make more sense given the query. Language understanding "finds" summarizations or captions and answers within your content, but unlike generative models like Azure OpenAI Service models GPT-3.5 or GPT-4, it does not create them. Only verbatim text from source documents is included in the response, which can then be rendered on a search results page for a more productive search experience.

State-of-the-art, pretrained models are used for summarization and ranking. To maintain the fast performance that users expect from search, semantic summarization and ranking are applied to just the top 50 results, as scored by the default scoring algorithm. Inputs are derived from the content in the search result. It cannot reach back to the search index to access other fields in the search document that were not returned in the query response. Inputs are subject to a token length of 8,960. These limits are necessary to maintain millisecond response times.

The default scoring algorithm is from Bing and Microsoft Research and integrated into the Azure AI Search infrastructure as an add-on feature. The models are used internally, are not exposed to the developer, and are nonconfigurable. For more information about the research and AI investments backing semantic ranker, see How AI from Bing is powering Azure AI Search (Microsoft Research Blog).

Semantic ranker also offers answers, captions, and highlighting within the response. For example, if the model classifies a query as a question and is 70% confident in the answer, the model returns a semantic answer. Additionally, semantic captions provide the most relevant content within the results and provide a brief snippet highlighting the most relevant words or phrases within that snippet.

Semantic ranker results are based off the data in the underlying search index, and the models provide relevance ranking and answers and captions based on the information retrieved from the index. Prior to using semantic ranker in a production environment, it is important to do further testing and to ensure that the dataset is accurate and appropriate for the intended use case. For more information and examples of how to evaluate semantic ranker, please see the content and appendix here.

System performance

In many AI systems, performance is often defined in relation to accuracy—that is, how often the AI system offers a correct prediction or output. With large-scale natural language models, two different users may look at the same output and have different opinions of how useful or relevant it is, which means that performance for these systems must be defined more flexibly. Here, we broadly consider performance to mean that the application performs as you and your users expect, including not generating harmful outputs.

Semantic ranker was trained on public content. As a result, the semantic relevance will vary based on the documents in the index and the queries issued against it. It is important to use your own judgment and research when you use this content for decision making.

Best practices for improving system performance

• Do spend time A/B testing your application with different query types, such as keyword versus hybrid plus semantic ranker. Figure out which query experience is best for your needs.
• Do expend a reasonable effort to set up your semantic configuration in accordance with the feature documentation.
• Do not trust the semantic answers if you do not have confidence in the accuracy of the information within the search index.
• Do not always trust semantic captions because they are extracted from customer content through a series of models that predicts the most relevant answers in a brief snippet.

Evaluation of Semantic ranker

Evaluation methods

Semantic ranker was evaluated through internal testing, including automated and human judgment on multiple datasets as well as feedback from internal customers. Testing includes the ranking of documents by scoring them as relevant or not relevant along with ranking documents in priority order of relevance. Likewise, the captions and answers functionality were also ranked via internal testing.

Evaluation results

We strive to ship all model updates regression-free (that is, the updated model should only improve the current production model). Each candidate is compared directly to the current production model by using metrics suitable for the feature being evaluated (for example, Normalized Discounted Cumulative Gain for ranking and precision/recall for answers). Semantic ranker models are trained, tuned, and evaluated by using a wide range of training data that is representative of documents that have different properties (language, length, formatting, styles, and tones) to support the broadest array of search scenarios. Our training and test data are drawn from:

Sources of documents:

• Academic and industry benchmarks
• Customer data (testing only, performed with customer permission)
• Synthetic data

Sources of queries:

• Benchmark query sets
• Customer-provided query sets (testing only, performed with customer permission)
• Synthetic query sets
• Human-generated query sets

Sources of labels for scoring query and document pairs:

• Academic and industry benchmark labels
• Customer labels (testing only, performed with customer permission)
• Synthetic data labels
• Human-scored labels

Evaluating and integrating semantic ranker for your use

The performance of semantic ranker varies depending on the real-world uses and conditions in which people use it. The quality of the relevance provided through the deep learning models that power semantic ranker capabilities is directly correlated with the data quality of your search index. For example, the models currently have token limitations that consider only the top 8,960 tokens for semantic answers. Therefore, if the semantic answer to a search query is found toward the end of a long document (beyond the 8,960 token limit), the answer will not be provided. The same rule applies for captions. Also, the semantic configuration lists relevant search fields in priority order. You can reorder the fields in this list to help tailor relevance to better suit your needs.

To ensure optimal performance in their scenarios, customers should conduct their own evaluations of the solutions they implement by using semantic ranker. Customers should generally follow an evaluation process that: (1) uses some internal stakeholders to evaluate results, (2) uses A/B experimentation to roll out semantic ranker to users, (3) incorporates KPIs and metrics monitoring when the service is deployed in experiences for the first time, and (4) tests and tweaks the semantic ranker configuration and/or index definition, including the surrounding experiences like user interface placement or business processes.

If you are developing an application in a high-stakes domain or industry, such as healthcare, human resources, education, or the legal field, assess how well the application works in your scenario, implement strong human oversight, evaluate how well users understand the limitations of the application, and comply with all relevant laws. Consider other mitigations based on your scenario.