Use textual summarization (preview)

Article
11/06/2024

This article shows you how to use textual summarization with Azure AI Video Indexer. This example uses Azure OpenAI model deployments.

Note

This feature only works with a standard account. Create a standard account.

Prerequisites

Review the overview of texual summarization.

Cloud

An Azure AI Video Indexer account connected to an Azure OpenAI account.
Access granted to Azure OpenAI in the desired Azure subscription. Currently, access to this service is granted only by application. You can apply for access to Azure OpenAI by completing the form.
An Azure OpenAI gpt-35-turbo or gpt-4 deployment. To benefit from keyframes based summaries you must select an Azure Open AI model that accepts visual input.. Read more here: Azure OpenAI Service models.
The Prompt Shields for direct attacks (jailbreak) filter should be added to the deployment. Read more here: Use content filters (preview) with Azure OpenAI Service.
It is recommended that you configure harmful content filters for categories such as “Violence,” “Hate,” “Sexual,” and “Self-harm.” While these filters are not mandatory, you should set them to either "Medium" or "Low" to filter out content of at least the Medium level of harmfullness. This setting ensures that content with a harmfulness rating of medium or higher is blocked. For increased safety, you may opt for a stricter setting. Once configured, please save the content filter settings.
A video uploaded to your Azure AI Video Indexer library.

Note

Azure AI Video Indexer is unable to connect to fine-tuned models.

Edge

Azure AI Video Indexer enabled by Arc allows you to use Azure AI Video Indexer on edge devices. The following are the technical requirements for this use case:

Video Indexer Enabled by Arc of version number 1.1.23 or higher.
Hardware requirements: GPU or Intel CPU. Running on CPU may be very slow and not recommended.We recommend using Nvidia V100 or higher. If there are multiple GPU nodes, you can add a node selector to select the desired node.
Using keyframes as part of the summary requires Nvidia A100 or higher.
Textual Video Summarization on Edge uses the Phi3.5 model only. For more hardware information please refer to the Phi 3.5 official release information.

Open web pages in your browser

It's easier to follow these instructions if you already have the needed web pages open. Copy and paste the following parameters to your favorite text editor. Here's a list to get you started:

VI account ID:
video file ID:
deployment name:
access token:

Open the Azure portal in one tab or window, and sign in.
1. Navigate to the Azure AI Video Indexer account page for the account ID.
2. Navigate to the Azure OpenAI account page
  1. From the menu, select Model deployments, then select Manage deployments. The Azure OpenAI Studio opens. Copy the Deployment Name you want to use.
Open the Azure AI Video Indexer web portal in another tab or window and sign in.
1. Navigate to the library page, choose a video, then right-click on it for the video file ID.
Open the Azure AI Video Indexer API in another tab or window and sign in.

Generate a text summary with or without keyframes

You can use the Azure AI Video Indexer web portal to summarize text.

If you don't have the web portal open already, open the Azure AI Video Indexer web portal.
Upload a file and wait for it to index.
Select the video to navigate to the media page.
If you want to use keyframes to produce the summary, select the Include visual keyframe insights to get better quality summary checkbox.
Select Generate summary. The textual summary is generated.

Note

If you aren't seeing the Generate summary button, you may be using a trial account. Create a standard account to use this feature.

Summary customizations

Once the summary is generated, you can make adjustments to it by customizing the summary style or changing the model deployment.

Customize summary style

You can also customize the summary by selecting the Customize summary icon. You can choose from Short, **Medium, or Long summary lengths, and you can choose Neutral, Casual, or Formal as a summary style. Once you make your choices, select Generate again.

Change model deployment

Select the Customize summary icon.
Select the model deployment you want to use from the Model deployment dropdown list.
Select Generate again.

Note

Remember that the model deployment is a combination of the model and the filters. Additionally, every time you customize the text summary it represents an API PUT request.

Create a video summary

Generate an access token

Generate an access token in the Azure portal:

Navigate to your Azure AI Video Indexer account.
From the menu, select Management then Management API.
From the Permission type dropdown, select Contributor.
From the Scope dropdown, select Account.
Select Generate. The access token is generated.
Copy the access token to your clipboard.

You should have all of the parameters needed to create a video summary ready to go in your text editor.

On the Azure AI Video Indexer API page, search for summarization.
Select the Try it button next to Create Video Summary. The API parameters pane opens.
Paste the access token on your clipboard into the accessToken field.
Select the location from the location dropdown list.
Copy and paste the VI account ID in the accountId field.
Copy the video file ID and paste it into the videoId field.
Choose the length of the summary, medium, short or long, from the length dropdown list.
Choose the style, neutral, casual, or formal, from the style dropdown list.
Select Send.

In the response, there's a summary ID. Copy that ID to your clipboard.

Get a video summary

First, check on the summary status. Until the summary job is finished, the state value shows "Processing," but there's no summary in the response. When the summary job is finished, the summary is displayed in the response.

Select GET Video Summary.
Enter the account ID, the video ID, the summary ID, and the access token into the appropriate fields.
Select Send.

Note

There is always a disclaimer available in the response about the summary. Also, there is a link to the job status API in the Location header.

You can list the video summaries by page. Select GET List Video Summaries and use the same parameters already described. If there's a large list of summaries, use the page parameters to return the response in pages. Use the summary ID for the GET Video Summary request.

Delete a video summary

Deleting the video summary is as simple as selecting DEL Delete Video Summary, using the same parameters, and selecting Send.

Error codes

Request	Status code	Error Message	Error Type	Reason	Mitigation

Create-Video-Summary	400	Account isn't connected to Azure OpenAI	INVALID_INPUT		Go to the Azure portal and connect an Account.
Create-Video-Summary	400	No deployment found with this name: {deploymentName}	INVALID_INPUT		Check the deployment name string.
Create-Video-Summary	400	This model doesn't support generating summaries.	INVALID_INPUT		Choose a different model
Create-Video-Summary	400	Couldn’t generate a summary because the AI model must have content filters added to avoid showing harmful content. " + $"Add the following policies: {string.Join(",", Enum.GetValues<FilterType>().Where(type => type != FilterType.JailBreak))}." + $"Set the AllowedContentLevel to at least {AllowedContentLevel.Medium} for both sources {FilterSource.Prompt} and {FilterSource.Completion}	SUMMARY_FILTERS_NOT_FOUND		See error message.
Create-Video-Summary	400	Video not found	VIDEO_NOT_FOUND
Create-Video-Summary	400	Video indexing hasn't finished successfully yet	VIDEO_ALREADY_IN_PROGRESS
Create-Video-Summary	400	Invalid summary length value '{length}'	INVALID_INPUT
Create-Video-Summary	400	Invalid summary style value '{style}	INVALID_INPUT
Get-Video-Summary	200	NA	NA
Get-Video-Summary	404	Summary '{summaryId}' not found for video '{videoId}'	NOT_FOUND
Get-Video-Summary	200	Couldn’t generate a summary because the video content was flagged as harmful. For details, see Content filters	NA	All sections in the video have triggered the content filter.	There's no solution for the specific video.
Get-Video-Summary	200	Not enough information in order to generate summary.	NA	The video insights don't contain enough information in order to create prompts for Azure OpenAI	If the preset of the video doesn't contain all the models, then reindexing with a more advanced preset that contains audio might help.
Get-Video-Summary	200	Couldn't get Azure OpenAI deployments for resource {_resourceName}
Get-Video-Summary	200	Deployment {DeploymentName} doesn't exist
Get-Video-Summary	200	Azure OpenAI couldn't be reached. Try running the summary again later.			Try summarization job again later.
Create-Video-Summary				No role assignment
Get-Video-Summary				No role assignment

Troubleshooting

Before looking for specific solutions, be sure that you created the resources in the Prerequisites section of the document.

Generate summary doesn't appear

You might not have connected your Azure OpenAI account to the Azure AI Video Indexer account correctly.

Couldn't generate summary

You need to add content filters to the deployment to avoid showing harmful content. Go back to the Azure OpenAI studio and add filters to your deployment.
You might have created a permissive content filter for content that has sensitive or harmful content. The content filter needs to be at least Medium. If you set the content filter to Medium AND harmful content remains in the video, the filter is triggered. In this case, there's no solution.
You might not have asked for a transcript or there might not be enough information generated during the indexing process.
You might be using an advanced preset.
The video might not have enough audio in it to create a transform.

Unable to use a model deployment or deployment missing

You might not have created a deployment.
Someone might have changed or deleted the deployed model.

Throttling

There might be too many requests being sent to VI. Wait for a few minutes and try again.

You receive the error "filter not found"

If you get a filter not found error, you'll get a detailed error describing what is missing. The error looks similar to this, but might have details about other missing things:

ErrorType":"SUMMARY_FILTERS_NOT_FOUND","Message":"Couldn't generate a summary because the model needs to be set with the right content filters to avoid showing harmful content. Input filter 'Jailbreak' must be enabled with action set to 'Annotate and block'. Trace id: '00000000-0000-0000-0000-000000000000'.

Note

The jailbreak filter is only required when using a keyframes based summary or a regular summary, as of November 1st, 2024.

To resolve the jailbreak issue:

Select Go to Azure Open AI studio.
Select Deployments.
Select the deployment you're working with.
Select Edit.
Configure the filters to have at least Medium level for each category.
Apply prompt shields for jailbreak attacks.
Select Create filter.

Other considerations

It's recommended to create the Azure AI Video Indexer and Azure OpenAI accounts in the same region or you might experience performance issues.

Share via

Use textual summarization (preview)

Prerequisites

Cloud

Edge

Open web pages in your browser

Generate a text summary with or without keyframes

Summary customizations

Customize summary style

Change model deployment

Create a video summary

Generate an access token

Get a video summary

Delete a video summary

Error codes

Troubleshooting

Generate summary doesn't appear

Couldn't generate summary

Unable to use a model deployment or deployment missing

Throttling

You receive the error "filter not found"

Other considerations

Feedback

Additional resources