How to use MedImageParse healthcare AI model for segmentation of medical images

In this article, you learn how to deploy MedImageParse as an online endpoint for real-time inference and issue a basic call to the API. The steps you take are:

• Deploy the model to a self-hosted managed compute.
• Grant permissions to the endpoint.
• Send test data to the model, receive, and interpret results.

MedImageParse - prompt-based segmentation of medical images

Biomedical image analysis is crucial for discovery in fields like cell biology, pathology, and radiology. Traditionally, tasks such as segmentation, detection, and recognition of relevant objects are addressed separately, which can limit the overall effectiveness of image analysis. However, MedImageParse unifies these tasks through image parsing, by jointly conducting segmentation, detection, and recognition across numerous object types and imaging modalities. By applying the interdependencies among these subtasks—such as the semantic labels of segmented objects—the model enhances accuracy and enables novel applications. For example, it allows users to segment all relevant objects in an image, by using a simple text prompt. This approach eliminates the need to manually specify bounding boxes for each object.

The following image shows the conceptual architecture of the MedImageParse model where an image embedding model is augmented with a task adaptation layer to produce segmentation masks and textual descriptions.

Remarkably, the segmentation masks and textual descriptions were achieved by using only standard segmentation datasets, augmented by natural-language labels, or descriptions harmonized with established biomedical object ontologies. This approach not only improved individual task performance but also offered an all-in-one tool for biomedical image analysis, paving the way for more efficient and accurate image-based biomedical discovery.

Prerequisites

To use the MedImageParse model, you need the following prerequisites:

A model deployment

Deployment to a self-hosted managed compute

MedImageParse model can be deployed to our self-hosted managed inference solution, which allows you to customize and control all the details about how the model is served. You can deploy the model through the catalog UI (in Azure AI Foundry or Azure Machine Learning studio) or deploy programmatically.

To deploy the model through the UI:

1. Go to the catalog.

2. Search for MedImageParse and select the model card.

3. On the model's overview page, select Deploy.

4. If given the option to choose between serverless API deployment and deployment using a managed compute, select Managed Compute.

5. Fill out the details in the deployment window.

   Note

   For deployment to a self-hosted managed compute, you must have enough quota in your subscription. If you don't have enough quota available, you can use our temporary quota access by selecting the option I want to use shared quota and I acknowledge that this endpoint will be deleted in 168 hours.

6. Select Deploy.

To deploy the model programmatically, see How to deploy and inference a managed compute deployment with code.

Work with a segmentation model

In this section, you consume the model and make basic calls to it.

Use REST API to consume the model

Consume the MedImageParse segmentation model as a REST API, using simple GET requests or by creating a client as follows:

from azure.ai.ml import MLClient
from azure.identity import DeviceCodeCredential

credential = DefaultAzureCredential()

ml_client_workspace = MLClient.from_config(credential)

In the deployment configuration, you get to choose authentication method. This example uses Azure Machine Learning token-based authentication. For more authentication options, see the corresponding documentation page. Also, note that the client is created from a configuration file that is created automatically for Azure Machine Learning virtual machines (VMs). Learn more on the corresponding API documentation page.

Make basic calls to the model

Once the model is deployed, use the following code to send data and retrieve segmentation masks.

import base64
import json
import os

sample_image_xray = os.path.join(image_path)

def read_image(image_path):
    with open(image_path, "rb") as f:
        return f.read()

sample_image =  "sample_image.png"
data = {
    "input_data": {
        "columns": [ "image", "text" ],
        "index": [ 0 ],
        "data": [
            [
                base64.encodebytes(read_image(sample_image)).decode("utf-8"),
                "neoplastic cells in breast pathology & inflammatory cells"
            ]
        ]
    }
}
data_json = json.dumps(data)

# Create request json
request_file_name = "sample_request_data.json"
with open(request_file_name, "w") as request_file:
    json.dump(data, request_file)

response = ml_client_workspace.online_endpoints.invoke(
    endpoint_name=endpoint_name,
    deployment_name=deployment_name,
    request_file=request_file_name,
)

Use MedImageParse REST API

MedImageParse model assumes a simple single-turn interaction where one request produces one response.

Request schema

Request payload is a JSON formatted string containing the following parameters:

The input_data object contains the following fields:

Request example

Requesting segmentation of all cells in a pathology image

{
  "input_data": {
    "columns": [
      "image",
      "text"
    ],
    "index":[0],
    "data": [
      ["iVBORw0KGgoAAAANSUhEUgAAAAIAAAACCAYAAABytg0kAAAAAXNSR0IArs4c6QAAAARnQU1BAACx\njwv8YQUAAAAJcEhZcwAAFiUAABYlAUlSJPAAAAAbSURBVBhXY/gUoPS/fhfDfwaGJe///9/J8B8A\nVGwJ5VDvPeYAAAAASUVORK5CYII=\n",
      "neoplastic & inflammatory cells "]
    ]
  }
}

Response schema

Response payload is a list of JSON-formatted strings, each corresponding to a submitted image. Each string contains a segmentation_object object.

segmentation_object contains the following fields:

segmentation_mask contains the following fields:

Response example

A simple inference requesting segmentation of two objects

[
  {
    "image_features": "{ 
    'data': '4oCwUE5HDQoa...',
    'shape': [2, 1024, 1024], 
    'dtype': 'uint8'}",
    "text_features": ['liver', 'pancreas']
  }
]

Supported image formats

The deployed model API supports images encoded in PNG format. For optimal results, we recommend using uncompressed/lossless PNGs with RGB images.

As described in the API specification, the model only accepts images in the resolution of 1024x1024pixels. Images need to be resized and padded (in the case of non-square aspect ratio).

See the Generating Segmentation for a Variety of Imaging Modalities notebook for techniques and sample code useful for submitting images of various sizes stored using various biomedical imaging formats.

Learn more from samples

MedImageParse is a versatile model that can be applied to a wide range of tasks and imaging modalities. For more examples see the following interactive Python Notebooks:

Getting started

• Deploying and Using MedImageParse: Learn how to deploy the MedImageParse model and integrate it into your workflow.

Advanced inferencing techniques and samples

• Generating Segmentation for a Variety of Imaging Modalities: Understand how to use MedImageParse to segment a wide variety of different medical images and learn some prompting techniques.

Related content

• CXRReportGen for grounded report generation
• MedImageInsight for grounded report generation