How to use Azure AI Search Index in Promptflow

Hi Seah Bryan,

Thanks for reaching out to Microsoft Q&A.

It sounds like you've successfully created the image search index using azure AI search and are now attempting to use it in Promptflow via the Index Lookup tool, but the results aren't coming back as expected. There are a few possible reasons for this mismatch between search explorer and Promptflow results.

Check the Search Query Format in Promptflow

• Even though you are using the same query, Promptflow might require a specific format or syntax when interacting with Azure AI Search. Ensure the query in Promptflow matches the expected format, particularly in how vectors are passed. For instance, vector queries should typically be in the form of a JSON object. Ensure that the embedding_value is correctly derived from your AI Vision vectorization output. #json

  { "vector": { "value": "<embedding_value>", "fields": ["embedding"] } }

Ensure Consistent Vector Encoding

• When using AI Vision vectorization for embedding images, ensure that the same encoding method is used consistently across both the Search Explorer and Promptflow. If there’s a mismatch in how vectors are encoded or stored, it could result in no matches.
• Double check that the vectors generated during indexing are correctly passed to the search function in Promptflow. You may want to output or log the vectors being sent in both environments to confirm they are identical.

Review Index Configuration in Promptflow

• Make sure that the Index Lookup tool in Promptflow is pointing to the correct index. Verify that the Index ID used in Promptflow matches the one used in the Search Explorer.
• Confirm that the fields in the index are being referenced correctly in Promptflow. Ensure that the correct field (ex: embedding or description) is being queried in the Lookup tool and that the index is configured to return the expected fields.

Embedding Size and Dimensionality

• Ensure that the vector dimension used during the query matches the dimension size of the embeddings stored in the index. If AI Vision produces a 512-dimension vector, for example, ensure the same vector size is being used in Promptflow.
• If the vector size or representation differs even slightly between the query and the indexed data, it can lead to no results being returned.

Index Refresh and Consistency

• Sometimes, newly indexed data may not be immediately searchable due to slight delays in propagation. Ensure the index is fully refreshed and ready to handle queries after you've embedded and indexed images.
• Try performing a few queries directly from the Search Explorer to confirm that all newly added data is searchable and consistent with your Promptflow queries.

Check Response Logging in Promptflow

• Log the raw responses from the Index Lookup tool in Promptflow. This can help you identify whether there are partial matches or specific errors that might explain the lack of results. You can enable logging within the Promptflow environment to capture details about the request and response cycles.

Use Filter Parameters (Optional)

• If your index contains filters (such as for filename or other metadata), ensure these filters are correctly applied in Promptflow as they are in Search Explorer. Sometimes differences in filter application might lead to no results being returned, even if the query matches.

Vector Matching Tolerance

• When using vector search, ensure the vector similarity metric or tolerance is configured correctly. If the similarity threshold is too strict, it might result in no matches being found. You can try lowering the threshold to capture more potential results in Promptflow.

Please feel free to click the 'Upvote' (Thumbs-up) button and 'Accept as Answer'. This helps the community by allowing others with similar queries to easily find the solution.