Tutorial: Upload images to the Bing Visual Search API
Warning
On October 30, 2020, the Bing Search APIs moved from Azure AI services to Bing Search Services. This documentation is provided for reference only. For updated documentation, see the Bing search API documentation. For instructions on creating new Azure resources for Bing search, see Create a Bing Search resource through the Azure Marketplace.
The Bing Visual Search API enables you to search the web for images similar to ones you upload. Use this tutorial to create a web application that can send an image to the API, and display the insights it returns within the webpage. Note that this application does not adhere to all Bing Use and Display Requirements for using the API.
You can find the full source code for this sample with additional error handling and annotations on GitHub.
The tutorial app illustrates how to:
- Upload an image to the Bing Visual Search API
- Display image search results in a web application
- Explore the different insights provided by the API
Prerequisites
Create an Azure resource
Start using the Bing Visual Search API by creating one of the following Azure resources:
- Available through the Azure portal until you delete the resource.
- Select the
S9pricing tier.
- Available through the Azure portal until you delete the resource.
- Use the same key and endpoint for your applications, across multiple Azure AI services.
Create and structure the webpage
Create an HTML page that sends an image to the Bing Visual Search API, receives insights, and displays them. In your favorite editor or IDE, create a file named "uploaddemo.html". Add the following basic HTML structure to the file:
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Visual Search Upload Demo</title>
</head>
<body>
</body>
</html>
Divide the page into a request section, where the user provides all the information required for the request, and a response section where the insights are displayed. Add the following <div> tags to the <body>. The <hr> tag visually separates the request section from the response section:
<div id="requestSection"></div>
<hr />
<div id="responseSection"></div>
Add a <script> tag to the <head> tag to contain the JavaScript for the application:
<script>
<\script>
Get the upload file
To let the user select an image to upload, the application uses the <input> tag with the type attribute set to file. The UI needs to make it clear that the application uses Bing to get the search results.
Add the following <div> to the requestSection <div>. The file input accepts a single file of any image type (for example, .jpg, .gif, .png). The onchange event specifies the handler that's called when a user selects a file.
The <output> tag is used to display a thumbnail of the selected image:
<div>
<p>Select image to get insights from Bing:
<input type="file" accept="image/*" id="uploadImage" name="files[]" size=40 onchange="handleFileSelect('uploadImage')" />
</p>
<output id="thumbnail"></output>
</div>
Create a file handler
Create a handler function that can read in the image you want to upload. While iterating through the files in the FileList object, the handler should make sure the selected file is an image file, and that its size is 1 MB or less. If the image is larger, you must reduce its size before uploading it. Lastly, the handler displays a thumbnail of the image:
function handleFileSelect(selector) {
var files = document.getElementById(selector).files; // A FileList object
for (var i = 0, f; f = files[i]; i++) {
// Ensure the file is an image file.
if (!f.type.match('image.*')) {
alert("Selected file must be an image file.");
document.getElementById("uploadImage").value = null;
continue;
}
// Image must be <= 1 MB and should be about 1500px.
if (f.size > 1000000) {
alert("Image must be less than 1 MB.");
document.getElementById("uploadImage").value = null;
continue;
}
var reader = new FileReader();
// Capture the file information.
reader.onload = (function(theFile) {
return function(e) {
var fileOutput = document.getElementById('thumbnail');
if (fileOutput.childElementCount > 0) {
fileOutput.removeChild(fileOutput.lastChild); // Remove the current pic, if it exists
}
// Render thumbnail.
var span = document.createElement('span');
span.innerHTML = ['<img class="thumb" src="', e.target.result,
'" title="', escape(theFile.name), '"/>'].join('');
fileOutput.insertBefore(span, null);
};
})(f);
// Read in the image file as a data URL.
reader.readAsDataURL(f);
}
}
Add and store a subscription key
The application requires a subscription key to make calls to the Bing Visual Search API. For this tutorial, you'll provide it in the UI. Add the following <input> tag (with the type attribute set to text) to the <body> just below the file's <output> tag:
<div>
<p>Subscription key:
<input type="text" id="key" name="subscription" size=40 maxlength="32" />
</p>
</div>
With the image and the subscription key, you can make the call to Bing Visual Search to get insights about the image. In this tutorial, the call uses the default market (en-us) and safe search value (moderate).
This application has an option to change these values. Add the following <div> below the subscription key <div>. The application uses a <select> tag to provide a drop-down list for market and safe search values. Both lists display the default value.
<div>
<p><a href="#" onclick="expandCollapse(options.id)">Query options</a></p>
<div id="options" style="display:none">
<p style="margin-left: 20px">Market:
<select id="mkt">
<option value="es-AR">Argentina (Spanish)</option>
<option value="en-AU">Australia (English)</option>
<option value="de-AT">Austria (German)</option>
<option value="nl-BE">Belgium (Dutch)</option>
<option value="fr-BE">Belgium (French)</option>
<option value="pt-BR">Brazil (Portuguese)</option>
<option value="en-CA">Canada (English)</option>
<option value="fr-CA">Canada (French)</option>
<option value="es-CL">Chile (Spanish)</option>
<option value="da-DK">Denmark (Danish)</option>
<option value="fi-FI">Finland (Finnish)</option>
<option value="fr-FR">France (French)</option>
<option value="de-DE">Germany (German)</option>
<option value="zh-HK">Hong Kong SAR(Traditional Chinese)</option>
<option value="en-IN">India (English)</option>
<option value="en-ID">Indonesia (English)</option>
<option value="it-IT">Italy (Italian)</option>
<option value="ja-JP">Japan (Japanese)</option>
<option value="ko-KR">Korea (Korean)</option>
<option value="en-MY">Malaysia (English)</option>
<option value="es-MX">Mexico (Spanish)</option>
<option value="nl-NL">Netherlands (Dutch)</option>
<option value="en-NZ">New Zealand (English)</option>
<option value="no-NO">Norway (Norwegian)</option>
<option value="zh-CN">People's Republic of China (Chinese)</option>
<option value="pl-PL">Poland (Polish)</option>
<option value="pt-PT">Portugal (Portuguese)</option>
<option value="en-PH">Philippines (English)</option>
<option value="ru-RU">Russia (Russian)</option>
<option value="ar-SA">Saudi Arabia (Arabic)</option>
<option value="en-ZA">South Africa (English)</option>
<option value="es-ES">Spain (Spanish)</option>
<option value="sv-SE">Sweden (Swedish)</option>
<option value="fr-CH">Switzerland (French)</option>
<option value="de-CH">Switzerland (German)</option>
<option value="zh-TW">Taiwan (Traditional Chinese)</option>
<option value="tr-TR">Türkiye (Turkish)</option>
<option value="en-GB">United Kingdom (English)</option>
<option value="en-US" selected>United States (English)</option>
<option value="es-US">United States (Spanish)</option>
</select>
</p>
<p style="margin-left: 20px">Safe search:
<select id="safesearch">
<option value="moderate" selected>Moderate</option>
<option value="strict">Strict</option>
<option value="off">off</option>
</select>
</p>
</div>
</div>
Add search options to the webpage
The application hides the lists in a collapsible <div> that's controlled by the Query options link. When you click the Query options link, the <div> expands so you can see and modify the query options. If you click the Query options link again, the <div> collapses and is hidden. The following snippet shows the Query options link's onclick handler. The handler controls whether the <div> is expanded or collapsed. Add this handler to the <script> section. The handler is used by all collapsible <div> sections in the demo.
// Contains the toggle state of divs.
var divToggleMap = {}; // divToggleMap['foo'] = 0; // 1 = show, 0 = hide
// Toggles between showing and hiding the specified div.
function expandCollapse(divToToggle) {
var div = document.getElementById(divToToggle);
if (divToggleMap[divToToggle] == 1) { // if div is expanded
div.style.display = "none";
divToggleMap[divToToggle] = 0;
}
else { // if div is collapsed
div.style.display = "inline-block";
divToggleMap[divToToggle] = 1;
}
}
Call the onclick handler
Add the following "Get insights" button below the options <div> in the body. The button enables you to initiate the call. When the button is clicked, the cursor is changed to the spinning wait cursor, and the onclick handler is called.
<p><input type="button" id="query" value="Get insights" onclick="document.body.style.cursor='wait'; handleQuery()" /></p>
Add the button's onclick handler, handleQuery() to the <script> tag.
Handle the query
The handler handleQuery() ensures the subscription key is present and is 32 characters long, and that an image is selected. It also clears any insights from a previous query. Afterwards, it calls the sendRequest() function to make the call.
function handleQuery() {
var subscriptionKey = document.getElementById('key').value;
// Make sure user provided a subscription key and image.
// For this demo, the user provides the key but typically you'd
// get it from secured storage.
if (subscriptionKey.length !== 32) {
alert("Subscription key length is not valid. Enter a valid key.");
document.getElementById('key').focus();
return;
}
var imagePath = document.getElementById('uploadImage').value;
if (imagePath.length === 0)
{
alert("Please select an image to upload.");
document.getElementById('uploadImage').focus();
return;
}
var responseDiv = document.getElementById('responseSection');
// Clear out the response from the last query.
while (responseDiv.childElementCount > 0) {
responseDiv.removeChild(responseDiv.lastChild);
}
// Send the request to Bing to get insights about the image.
var f = document.getElementById('uploadImage').files[0];
sendRequest(f, subscriptionKey);
}
Send the search request
The sendRequest() function formats the endpoint URL, sets the Ocp-Apim-Subscription-Key header to the subscription key, appends the binary of the image to upload, specifies the response handler, and makes the call:
function sendRequest(file, key) {
var market = document.getElementById('mkt').value;
var safeSearch = document.getElementById('safesearch').value;
var baseUri = `https://api.cognitive.microsoft.com/bing/v7.0/images/visualsearch?mkt=${market}&safesearch=${safeSearch}`;
var form = new FormData();
form.append("image", file);
var request = new XMLHttpRequest();
request.open("POST", baseUri);
request.setRequestHeader('Ocp-Apim-Subscription-Key', key);
request.addEventListener('load', handleResponse);
request.send(form);
}
Get and handle the API response
The handleResponse() function handles the response from the call to Bing Visual Search. If the call succeeds, it parses the JSON response into the individual tags, which contain the insights. Next, it adds the search results to the page. The application then creates a collapsible <div> for each tag to manage how much data is displayed. Add the handler to the <script> section.
function handleResponse() {
if(this.status !== 200){
alert("Error calling Bing Visual Search. See console log for details.");
console.log(this.responseText);
return;
}
var tags = parseResponse(JSON.parse(this.responseText));
var h4 = document.createElement('h4');
h4.textContent = 'Bing internet search results';
document.getElementById('responseSection').appendChild(h4);
buildTagSections(tags);
document.body.style.cursor = 'default'; // reset the wait cursor set by query insights button
}
Parse the response
The parseResponse function converts the JSON response into a dictionary object by iterating through json.tags.
function parseResponse(json) {
var dict = {};
for (var i =0; i < json.tags.length; i++) {
var tag = json.tags[i];
if (tag.displayName === '') {
dict['Default'] = JSON.stringify(tag);
}
else {
dict[tag.displayName] = JSON.stringify(tag);
}
}
return(dict);
}
Build a tag section
The buildTagSections() function iterates through the parsed JSON tags and calls the buildDiv() function to build a <div> for each tag. Each tag is displayed as a link. When the link is clicked, the tag expands, showing the insights associated with the tag. Clicking the link again causes the section to collapse.
function buildTagSections(tags) {
for (var tag in tags) {
if (tags.hasOwnProperty(tag)) {
var tagSection = buildDiv(tags, tag);
document.getElementById('responseSection').appendChild(tagSection);
}
}
}
function buildDiv(tags, tag) {
var tagSection = document.createElement('div');
tagSection.setAttribute('class', 'subSection');
var link = document.createElement('a');
link.setAttribute('href', '#');
link.setAttribute('style', 'float: left;')
link.text = tag;
tagSection.appendChild(link);
var contentDiv = document.createElement('div');
contentDiv.setAttribute('id', tag);
contentDiv.setAttribute('style', 'clear: left;')
contentDiv.setAttribute('class', 'section');
tagSection.appendChild(contentDiv);
link.setAttribute('onclick', `expandCollapse("${tag}")`);
divToggleMap[tag] = 0; // 1 = show, 0 = hide
addDivContent(contentDiv, tag, tags[tag]);
return tagSection;
}
Display the search results in the webpage
The buildDiv() function calls the addDivContent function to build the contents of each tag's collapsible <div>.
A tag's content includes the JSON from the response for the tag. Initially, only the first 100 characters of the JSON are shown, but you can click the JSON string to show all the JSON. If you click it again, the JSON string collapses back to 100 characters.
Next, add the action types found in the tag. For each action type, call the appropriate functions to add its insights:
function addDivContent(div, tag, json) {
// Adds the first 100 characters of the json that contains
// the tag's data. The user can click the text to show the
// full json. They can click it again to collapse the json.
var para = document.createElement('p');
para.textContent = String(json).substr(0, 100) + '...';
para.setAttribute('title', 'click to expand');
para.setAttribute('style', 'cursor: pointer;')
para.setAttribute('data-json', json);
para.addEventListener('click', function(e) {
var json = e.target.getAttribute('data-json');
if (e.target.textContent.length <= 103) { // 100 + '...'
e.target.textContent = json;
para.setAttribute('title', 'click to collapse');
}
else {
para.textContent = String(json).substr(0, 100) + '...';
para.setAttribute('title', 'click to expand');
}
});
div.appendChild(para);
var parsedJson = JSON.parse(json);
// Loop through all the actions in the tag and display them.
for (var j = 0; j < parsedJson.actions.length; j++) {
var action = parsedJson.actions[j];
var subSectionDiv = document.createElement('div');
subSectionDiv.setAttribute('class', 'subSection');
div.appendChild(subSectionDiv);
var h4 = document.createElement('h4');
h4.innerHTML = action.actionType;
subSectionDiv.appendChild(h4);
if (action.actionType === 'ImageResults') {
addImageWithWebSearchUrl(subSectionDiv, parsedJson.image, action);
}
else if (action.actionType === 'DocumentLevelSuggestions') {
addRelatedSearches(subSectionDiv, action.data.value);
}
else if (action.actionType === 'RelatedSearches') {
addRelatedSearches(subSectionDiv, action.data.value);
}
else if (action.actionType === 'PagesIncluding') {
addPagesIncluding(subSectionDiv, action.data.value);
}
else if (action.actionType === 'VisualSearch') {
addRelatedImages(subSectionDiv, action.data.value);
}
else if (action.actionType === 'Recipes') {
addRecipes(subSectionDiv, action.data.value);
}
else if (action.actionType === 'ShoppingSources') {
addShopping(subSectionDiv, action.data.offers);
}
else if (action.actionType === 'ProductVisualSearch') {
addProducts(subSectionDiv, action.data.value);
}
else if (action.actionType === 'TextResults') {
addTextResult(subSectionDiv, action);
}
else if (action.actionType === 'Entity') {
addEntity(subSectionDiv, action);
}
}
}
Display insights for different actions
The following functions display insights for different actions. The functions either provide a clickable image or clickable link that sends you to a webpage with more information about the image. This page is either hosted by Bing.com or the image's original website. Not all of the insights' data is displayed in this application. To see all the fields available for an insight, see the Images - Visual Search reference.
Note
There's a minimum amount of insight information you must display in the page. See the Bing Search API use and display requirements for more.
RelatedImages insights
The addRelatedImages() function creates a title for each of the websites hosting the related image by iterating through the list of RelatedImages actions, and appending an <img> tag to the outside <div> for each:
function addRelatedImages(div, images) {
var length = (images.length > 10) ? 10 : images.length;
// Set the title to the website that hosts the image. The title displays
// when the user hovers over the image.
// Make the image clickable. If the user clicks the image, they're taken
// to the image in Bing.com.
for (var j = 0; j < length; j++) {
var img = document.createElement('img');
img.setAttribute('src', images[j].thumbnailUrl + '&w=120&h=120');
img.setAttribute('style', 'margin: 20px 20px 0 0; cursor: pointer;');
img.setAttribute('title', images[j].hostPageDisplayUrl);
img.setAttribute('data-webSearchUrl', images[j].webSearchUrl)
img.addEventListener('click', function(e) {
var url = e.target.getAttribute('data-webSearchUrl');
window.open(url, 'foo');
})
div.appendChild(img);
}
}
PagesIncluding insights
The addPagesIncluding() function creates a link for each of the websites hosting the uploaded image by iterating through the list of PagesIncluding actions, and appending an <img> tag to the outside <div> for each:
// Display links to the first 5 webpages that include the image.
// TODO: Add 'more' link in case the user wants to see all of them.
function addPagesIncluding(div, pages) {
var length = (pages.length > 5) ? 5 : pages.length;
for (var j = 0; j < length; j++) {
var page = document.createElement('a');
page.text = pages[j].name;
page.setAttribute('href', pages[j].hostPageUrl);
page.setAttribute('style', 'margin: 20px 20px 0 0');
page.setAttribute('target', '_blank')
div.appendChild(page);
div.appendChild(document.createElement('br'));
}
}
RelatedSearches insights
The addRelatedSearches() function creates a link for the website hosting the image, by iterating through the list of RelatedSearches actions and appending an <img> tag to the outside <div> for each:
// Display the first 10 related searches. Include a link with the image
// that when clicked, takes the user to Bing.com and displays the
// related search results.
// TODO: Add 'more' link in case the user wants to see all of them.
function addRelatedSearches(div, relatedSearches) {
var length = (relatedSearches.length > 10) ? 10 : relatedSearches.length;
for (var j = 0; j < length; j++) {
var childDiv = document.createElement('div');
childDiv.setAttribute('class', 'stackLink');
div.appendChild(childDiv);
var img = document.createElement('img');
img.setAttribute('src', relatedSearches[j].thumbnail.url + '&w=120&h=120');
img.setAttribute('style', 'margin: 20px 20px 0 0;');
childDiv.appendChild(img);
var relatedSearch = document.createElement('a');
relatedSearch.text = relatedSearches[j].displayText;
relatedSearch.setAttribute('href', relatedSearches[j].webSearchUrl);
relatedSearch.setAttribute('target', '_blank');
childDiv.appendChild(relatedSearch);
}
}
Recipes insights
The addRecipes() function creates a link for each of recipes returned by iterating through the list of Recipes actions, and appending an <img> tag to the outside <div> for each:
// Display links to the first 10 recipes. Include the recipe's rating,
// if available.
// TODO: Add 'more' link in case the user wants to see all of them.
function addRecipes(div, recipes) {
var length = (recipes.length > 10) ? 10 : recipes.length;
for (var j = 0; j < length; j++) {
var para = document.createElement('p');
var recipe = document.createElement('a');
recipe.text = recipes[j].name;
recipe.setAttribute('href', recipes[j].url);
recipe.setAttribute('style', 'margin: 20px 20px 0 0');
recipe.setAttribute('target', '_blank')
para.appendChild(recipe);
if (recipes[j].hasOwnProperty('aggregateRating')) {
var span = document.createElement('span');
span.textContent = 'rating: ' + recipes[j].aggregateRating.text;
para.appendChild(span);
}
div.appendChild(para);
}
}
Shopping insights
The addShopping() function creates a link for any returned shopping results by iterating through the list of RelatedImages actions, and appending an <img> tag to the outside <div> for each:
// Display links for the first 10 shopping offers.
// TODO: Add 'more' link in case the user wants to see all of them.
function addShopping(div, offers) {
var length = (offers.length > 10) ? 10 : offers.length;
for (var j = 0; j < length; j++) {
var para = document.createElement('p');
var offer = document.createElement('a');
offer.text = offers[j].name;
offer.setAttribute('href', offers[j].url);
offer.setAttribute('style', 'margin: 20px 20px 0 0');
offer.setAttribute('target', '_blank')
para.appendChild(offer);
var span = document.createElement('span');
span.textContent = 'by ' + offers[j].seller.name + ' | ' + offers[j].price + ' ' + offers[j].priceCurrency;
para.appendChild(span);
div.appendChild(para);
}
}
Products insights
The addProducts() function creates a link for any returned products results by iterating through the list of Products actions, and appending an <img> tag to the outside <div> for each:
// Display the first 10 related products. Display a clickable image of the
// product that takes the user to Bing.com search results for the product.
// If there are any offers associated with the product, provide links to the offers.
// TODO: Add 'more' link in case the user wants to see all of them.
function addProducts(div, products) {
var length = (products.length > 10) ? 10 : products.length;
for (var j = 0; j < length; j++) {
var childDiv = document.createElement('div');
childDiv.setAttribute('class', 'stackLink');
div.appendChild(childDiv);
var img = document.createElement('img');
img.setAttribute('src', products[j].thumbnailUrl + '&w=120&h=120');
img.setAttribute('title', products[j].name);
img.setAttribute('style', 'margin: 20px 20px 0 0; cursor: pointer;');
img.setAttribute('data-webSearchUrl', products[j].webSearchUrl)
img.addEventListener('click', function(e) {
var url = e.target.getAttribute('data-webSearchUrl');
window.open(url, 'foo');
})
childDiv.appendChild(img);
if (products[j].insightsMetadata.hasOwnProperty('aggregateOffer')) {
if (products[j].insightsMetadata.aggregateOffer.offerCount > 0) {
var offers = products[j].insightsMetadata.aggregateOffer.offers;
// Show all the offers. Not all markets provide links to offers.
for (var i = 0; i < offers.length; i++) {
var para = document.createElement('p');
var offer = document.createElement('a');
offer.text = offers[i].name;
offer.setAttribute('href', offers[i].url);
offer.setAttribute('style', 'margin: 20px 20px 0 0');
offer.setAttribute('target', '_blank')
para.appendChild(offer);
var span = document.createElement('span');
span.textContent = 'by ' + offers[i].seller.name + ' | ' + offers[i].price + ' ' + offers[i].priceCurrency;
para.appendChild(span);
childDiv.appendChild(para);
}
}
else { // Otherwise, just show the lowest price that Bing found.
var offer = products[j].insightsMetadata.aggregateOffer;
var para = document.createElement('p');
para.textContent = `${offer.name} | ${offer.lowPrice} ${offer.priceCurrency}`;
childDiv.appendChild(para);
}
}
}
}
TextResult insights
The addTextResult() function displays any text that was recognized in the image:
function addTextResult(div, action) {
var text = document.createElement('p');
text.textContent = action.displayName;
div.appendChild(text);
}
The addEntity() function displays a link that takes the user to Bing.com where they can get details about the entity type in the image, if any was detected:
// If the image is of a person, the tag might include an entity
// action type. Display a link that takes the user to Bing.com
// where they can get details about the entity.
function addEntity(div, action) {
var entity = document.createElement('a');
entity.text = action.displayName;
entity.setAttribute('href', action.webSearchUrl);
entity.setAttribute('style', 'margin: 20px 20px 0 0');
entity.setAttribute('target', '_blank');
div.appendChild(entity);
}
The addImageWithWebSearchUrl() function displays a clickable image to the <div> that takes the user to search results on Bing.com:
function addImageWithWebSearchUrl(div, image, action) {
var img = document.createElement('img');
img.setAttribute('src', image.thumbnailUrl + '&w=120&h=120');
img.setAttribute('style', 'margin: 20px 20px 0 0; cursor: pointer;');
img.setAttribute('data-webSearchUrl', action.webSearchUrl);
img.addEventListener('click', function(e) {
var url = e.target.getAttribute('data-webSearchUrl');
window.open(url, 'foo');
})
div.appendChild(img);
}
Add a CSS style
Add the following <style> section to the <head> tag to organize the layout of the webpage:
<style>
.thumb {
height: 75px;
border: 1px solid #000;
}
.stackLink {
width:180px;
min-height:210px;
display:inline-block;
}
.stackLink a {
float:left;
clear:left;
}
.section {
float:left;
display:none;
}
.subSection {
clear:left;
float:left;
}
</style>