How to use Azure Face API on RTSP streaming?

Hello Harshita Gupta,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that you are utilizing the Azure Face API to analyze RTSP video streams. While the API provides accurate face detection results, it has issues from high latency, leading to screen freezes during streaming.

Solution

As a solution architect the solution designed have to balance accuracy and latency, but achieving optimal results depends on several factors, including network speed, the complexity of frames, and processing power. However, if you are sure that Frame Size, API Parameters, and Frame Quality is okay, because that's my components for accuracy in this context. I will provide you with tips to reduce latency, implementation with sample code and more optimization strategy.

Tips for Reducing Latency

1. Limit the frame rate from the RTSP stream if possible.
2. Resize frames to a smaller size before sending to the API.
3. Use asynchronous requests to process multiple frames in parallel.
4. If possible, run initial face detection locally and use Azure Face API for more detailed analysis.

Implementation Sample Code

Here I provide a Python-based solution that minimizes latency without compromising the accuracy of face detection for you with implementation focusing on asynchronous processing and reduced frame size:

import cv2
import requests
import asyncio
import aiohttp
import numpy as np
# Azure Face API endpoint and key
face_api_url = "https://<your-face-api-endpoint>.cognitiveservices.azure.com/face/v1.0/detect"
subscription_key = "<your-subscription-key>"
# Headers for the API request
headers = {
    'Ocp-Apim-Subscription-Key': subscription_key,
    'Content-Type': 'application/octet-stream'
}
async def detect_faces(session, frame):
    _, img_encoded = cv2.imencode('.jpg', frame)
    try:
        async with session.post(face_api_url, headers=headers, data=img_encoded.tobytes(), params={
            'returnFaceId': 'true',
            'returnFaceLandmarks': 'false',
            'returnFaceAttributes': 'age,gender,smile,facialHair,glasses,emotion'
        }) as response:
            if response.status == 200:
                return await response.json()
            else:
                return None
    except Exception as e:
        print(f"Error: {e}")
        return None
async def process_stream(rtsp_url):
    cap = cv2.VideoCapture(rtsp_url)
    async with aiohttp.ClientSession() as session:
        while True:
            ret, frame = cap.read()
            if not ret:
                break
            # Resize the frame to reduce latency
            frame = cv2.resize(frame, (640, 480))
            # Run face detection asynchronously
            faces = await detect_faces(session, frame)
            # Draw face rectangles
            if faces:
                for face in faces:
                    rect = face['faceRectangle']
                    cv2.rectangle(frame, (rect['left'], rect['top']),
                                  (rect['left'] + rect['width'], rect['top'] + rect['height']),
                                  (0, 255, 0), 2)
            cv2.imshow('Frame', frame)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
    cap.release()
    cv2.destroyAllWindows()
# RTSP stream URL
rtsp_url = "rtsp://your-stream-url"
# Run the stream processing
asyncio.run(process_stream(rtsp_url))

The above implementation is a kind of solution that will provide a good balance between accuracy and reduced latency for processing RTSP streams with Azure Face API using Python. Adjusting the frame size, rate, and leveraging asynchronous processing will help mitigate latency issues.

Lastly, Optimization Strategies

1. Deploy edge computing solutions to preprocess and filter frames before sending them to the cloud.
2. If the latency is still high, consider sending frames in batches to reduce the number of API calls.
3. Process multiple RTSP streams in parallel using multithreading or multiprocessing, depending on your hardware capabilities.

References

Source: Async IO in Python. Accessed, 7/22/2024.

Source: OpenCV VideoCapture Class.Accessed, 7/22/2024.

Source: Real-Time Video Processing with OpenCV and Python. Accessed, 7/22/2024.

Source: Reducing Video Latency. Accessed, 7/22/2024.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Hello @Harshita Gupta

Thanks for reaching out to us, you may encounter latency when using the Face service. Latency refers to any kind of delay that occurs when systems communicate over a network. In general, possible causes of latency include:

• The physical distance each packet must travel from source to destination.
• Problems with the transmission medium.
• Errors in routers or switches along the transmission path.
• The time required by antivirus applications, firewalls, and other security mechanisms to inspect packets.
• Malfunctions in client or server applications.

This section describes how you can mitigate various causes of latency specific to the Azure AI Face service.

Choose the appropriate region for your Face resource

The network latency, the time it takes for information to travel from source (your application) to destination (your Azure resource), is strongly affected by the geographical distance between the application making requests and the Azure server responding to those requests. For example, if your Face resource is located in EastUS, it has a faster response time for users in New York, and users in Asia experience a longer delay.

We recommend that you select a region that is closest to your users to minimize latency. If your users are distributed across the world, consider creating multiple resources in different regions and routing requests to the region nearest to your customers. Alternatively, you may choose a region that is near the geographic center of all your customers.

The document you may be interested in here-

https://zcusa.951200.xyz/en-us/azure/ai-services/computer-vision/how-to/mitigate-latency

If you have a specific case that you were experience long latency, I am happy to enable you a free ticket to engage with support team for more backend data inverstigation.

I hope this helps, please let us know.

Regards,

Yutong

-Please kindly accept the answer if you feel helpful to support the community, thanks a lot.