{"id":3251,"date":"2017-07-01T01:56:54","date_gmt":"2017-07-01T08:56:54","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=3251"},"modified":"2020-03-14T20:22:07","modified_gmt":"2020-03-15T03:22:07","slug":"building-image-classification-pipeline-using-serverless-architecture","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/building-image-classification-pipeline-using-serverless-architecture\/","title":{"rendered":"Building an Image Classification Pipeline Using Serverless Architecture"},"content":{"rendered":"

Background<\/h2>\n

The International Committee of the Red Cross<\/a> (ICRC)\u00a0Trace-the-Face<\/a>\u00a0program uses photos to match up missing family members who have been separated due to migration and conflict. We have been working with the ICRC to update and extend Trace-the-Face to perform automatic face detection and matching using machine learning, via Microsoft Cognitive Services Face API<\/a>.<\/p>\n

As we discussed how Trace-the-Face could use computer vision technologies, the ICRC grew more and more intrigued and developed a long list of their challenges that could potentially be solved by the use of computer vision technologies. Through a series of workshops and discussions, we designed a centralized image classification pipeline that could be integrated across the organization to build up a rich, tagged image set over time \u2013 including faces found in any images from any source \u2013 with the intent of addressing many of their computer vision related challenges.<\/p>\n

The Solution<\/h2>\n

In February, ICRC and Microsoft started a collaboration on Imaginem<\/a>, an image classification pipeline<\/strong> built on top of Azure Functions<\/a>. If you’re not familiar with Azure Functions, it is a framework for building serverless microservices that can easily be deployed to the cloud. Azure Functions is built on top of Azure App Service<\/a>, which means it’s easily manageable and has built-in scaling. Visual Studio supports developing Azure Functions, but you can easily use your favorite text editor to write either C# or JavaScript files.<\/p>\n

The choice to go serverless gave us the flexibility to add or remove functions as the pipeline grows. The initial focus was to do general image classification and face detection and matching. \u00a0We have additional plans to incorporate custom image classification and object detection built with ML frameworks like CNTK<\/a> and TensorFlow<\/a>.<\/p>\n

Our architecture uses message queuing to move images along the pipeline. Each message contains the information needed to move the image to the next step, \u00a0including a link to the image blob and the collected properties about the image classification.<\/p>\n

\"\"<\/h4>\n

The Code<\/h3>\n

Each step in the pipeline is an Azure Function<\/a>;\u00a0building the pipeline involved creating each Azure Function, and then adding its input queue name into the pipeline definition.<\/p>\n

Initially, you can set these steps pre-deployment<\/a> in the ARM deployment parameters<\/a>.<\/p>\n

    \"pipelineDefinition\": {\r\n      \"value\": \"generalclassification,ocr,facedetection,facecrop,faceprint,facematch,pipelineoutput\"\r\n    },<\/pre>\n
If you set the steps post-deployment, you can simply edit them in your Function App’s environment variables<\/a>.<\/p>\n
Below is a simple function that breaks down a message, retrieves the image from Azure Blob Storage<\/a> and then sends it into Microsoft Cognitive Services Computer Vision API<\/a> for image tagging. This example is written in C#, but you could also write it using JavaScript<\/a>.<\/p>\n
#load \"..\\Common\\FunctionHelper.csx\"\r\n#load \"..\\Common\\ComputerVisionFunctions.csx\"\r\n\r\nusing System.Net.Http.Headers;\r\nusing System.Text;\r\nusing System.Net.Http;\r\nusing System.Web;\r\nusing System.Runtime;\r\nusing Newtonsoft.Json;\r\n\r\nprivate const string ClassifierName = \"generalclassification\";\r\n\r\npublic static void Run(string inputMsg, TraceWriter log)\r\n{\r\n    log.Info(inputMsg);\r\n    PipelineHelper.Process(GeneralClassificationFunction, ClassifierName, inputMsg, log);\r\n}\r\n\r\npublic static dynamic GeneralClassificationFunction(dynamic inputJson, string imageUrl, TraceWriter log)\r\n{\r\n    var parameters = inputJson.job_definition.image_parameters;\r\n    var response = ComputerVisionFunctions.AnalyzeImageAsync(imageUrl, log).Result;\r\n    return JsonConvert.DeserializeObject(response);\r\n}<\/pre>\n
The code includes calls to helper methods\u00a0that allow the function to take part in the pipeline in a well-defined way. They ensure adherence to the message contract and then advance the message to the next defined queue. These methods\u00a0are defined in the GitHub repo<\/a>, if you want to see what they do in the context of the pipeline.<\/p>\n
The first parameter in the function’s Run<\/em>\u00a0method is defined as string inputMsg<\/span>\u00a0and is tied to an Azure Function,\u00a0QueueTrigger<\/a>. The framework will watch the Azure Storage Queue that is defined in the function.json file for new messages and trigger the function when a new message is found.<\/p>\n
Here is a sample of a function definition for a QueueTrigger<\/em> from the documentation<\/a>:<\/p>\n
{\r\n    \"type\": \"queueTrigger\",\r\n    \"direction\": \"in\",\r\n    \"name\": \"<The name used to identify the trigger data in your code>\",\r\n    \"queueName\": \"<Name of queue to poll>\",\r\n    \"connection\":\"<Name of app setting - see below>\"\r\n}<\/pre>\n
To add a custom step to the pipeline:<\/p>\n
    \n
  1. Create a new Azure Storage Queue in the attached Storage Account for your Function App<\/li>\n
  2. Add a new function to the Function App with a QueueTrigger<\/em><\/li>\n
  3. Set the name<\/span>\u00a0\u00a0field to “inputMsg” to match our\u00a0Run<\/em>\u00a0method<\/li>\n
  4. Set the queueName<\/span>\u00a0\u00a0field to the name of the input queue that you created in step 1<\/li>\n
  5. Set the connection<\/span>\u00a0\u00a0field to the Storage Account name defined in appsettings.json<\/em><\/li>\n<\/ol>\n
    Keep in mind that you can perform any action on an image in this step of the pipeline. For example, we’ve had partners use a step to combine multiple images into a grid for submission to the Cognitive Services Face API in order to save on API calls.<\/p>\n
    Conclusion<\/h2>\n
    Through our collaboration with ICRC, we created a nice, tidy reusable framework for processing images using message queues. It detects faces, searches for similar or matching faces, identifies objects, and extracts text characters from images. The results can be easily extracted from the JSON object or retrieved from the SQL database.<\/p>\n
    Imaginem could be reused in many scenarios that require facial recognition and matching.\u00a0 We welcome your feedback and PRs!<\/p>\n
    Note<\/strong>: This code was built using preview tools for Azure Functions in Visual Studio 2015. The Azure Functions team recommends using Visual Studio 2017 (read more<\/a>).<\/p>\n
    Resources<\/h4>\n