"Import and vectorize data" -- vector or hybrid index?

Barry Briggs 80

When using the "Import and Vectorize Data" option (say, on JSON documents), are the original text fields preserved in the index with vectors added (for example, "body:" and "bodyVector:") or is the document simply vectorized as a whole? If the latter, is it possible in some way to extract specific fields in Azure OpenAI for example for citations?

Laxman Reddy Revuri 1,525 Reputation points Microsoft Vendor

2024-12-23T00:46:24.9066667+00:00

Hi @Barry Briggs
Thanks for the question and using MS Q&A platform.
Hybrid Index: The explanation also stands as to how a hybrid index supports both original text fields e.g. body and vectorized versions e.g. bodyVector as storage constructs. This configuration makes it possible for the users to use raw text in keyword searches as well as vectorized fields in similarity searches.
https://zcusa.951200.xyz/en-us/azure/search/hybrid-search-overview
Vector-Only Index: The description of a vector-only index is also accurate. In this instance, however, all original text fields are lost as only the vector representation of the whole document is kept. This reduces the querying option to standard similarity searches based on the vectors but does not allow direct extraction of fields.
https://zcusa.951200.xyz/en-us/azure/search/vector-search-overview
Field Extraction for Citations: The suggested techniques for extracting certain fields likebody for the purpose of citations are reasonable. Indexing original fields for example separately, citing other forms of retrieval, such as a hybrid index, are also effective approaches to ensure particular text that needs to be searched and retrieved is available. https://zcusa.951200.xyz/en-us/azure/search/vector-search-how-to-configure-vectorizer
I hope this information is helpful.
Laxman Reddy Revuri 1,525 Reputation points Microsoft Vendor

2024-12-24T05:59:01.64+00:00

Hi @Barry Briggs
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Barry Briggs 80 Reputation points

2024-12-24T15:28:01.33+00:00

Not exactly. I'm curious what exactly happens when the wizard is selected. In my case I have a set of JSON documents with fields like title, author, body, etc. After the wizard indexing runs there is an index created -- is that a purely vector index or is it a hybrid index?

Empirically it seems it is a hybrid index -- when loaded into the Azure OpenAI chat playground the index shows as hybrid; and when used it seems that the text /keyword fields are preserved (which is good!).

Just looking for confirmation that my assumptions are correct. Thanks very much!

Accepted answer

Laxman Reddy Revuri 1,525 Reputation points Microsoft Vendor

2024-12-26T04:09:53.2833333+00:00

Hi @Barry Briggs
You are correct! Using the "Import and Vectorize Data" wizard in Azure Cognitive Search with a dataset like JSON documents containing fields such as title, author, and body results in the creation of a hybrid index.
Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.
Please sign in to rate this answer.

1 person found this answer helpful.
Laxman Reddy Revuri 1,525 Reputation points Microsoft Vendor

2024-12-27T03:55:46.64+00:00

Hi @Barry Briggs
Following up to see if you have chance to check my previous response and help us with requested information to check and assist you further on this.

Alan José Vladimir Mendoza Navarro 0 Reputation points

2025-01-03T02:24:43.93+00:00

Hi @Laxman Reddy Revuri , I have a question that might be related, when I create an index with the import and vectorize data wizard using Cosmos DB for NO SQl as a data source from the Azure Portal, at the beginning everything is OK, that is, it tells me which field I want to vectorize and all that, but when I have other additional fields that are not vectors it does not give me the option to choose whether they are filterable, sortable, and facetable, is that something that could be corrected or configured in a certain way? I was looking for documentation and I can't find it until now, thank you very much.
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

"Import and vectorize data" -- vector or hybrid index?

0 additional answers

Your answer