{"id":3726,"date":"2017-07-31T11:17:44","date_gmt":"2017-07-31T18:17:44","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=3726"},"modified":"2020-03-19T14:06:01","modified_gmt":"2020-03-19T21:06:01","slug":"using-object-detection-complex-image-classification-scenarios","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/using-object-detection-complex-image-classification-scenarios\/","title":{"rendered":"Using Object Detection for Complex Image Classification Scenarios"},"content":{"rendered":"

We recently worked with\u00a0SMART Business<\/a>, a Ukrainian consulting services company,\u00a0along with their partner, a large manufacturer of confectionery products in Central & Eastern Europe, to build a machine learning model which validates whether distributors are stocking chocolates correctly.<\/p>\n

This code story provides an overview of different image classification approaches for various levels of complexity that we explored while developing our solution.<\/p>\n

Background<\/h2>\n

The company we worked with has a huge distribution network of supermarket chains across over fourteen countries. Each of these distributors is required to arrange chocolates on their stands according to standardized policies. Each policy describes what shelf a given chocolate should be on and in what order it should be stocked.<\/p>\n

There are huge costs associated with \u201croutine\u201d audit activities to enforce these policies. SMART Business\u00a0wanted to develop a system in which an auditor or store manager could take a picture and be told immediately whether the shelf was stocked correctly, like in the image below.<\/p>\n

\"example
Valid policy (left), invalid policy (right)<\/figcaption><\/figure><\/p>\n

Investigation<\/h2>\n

During our scoping, we investigated a couple of approaches to image classification including Microsoft’s Custom Vision Service<\/a>, Transfer Learning using CNTK ResNet<\/a>, and Object Detection with CNTK Fast-RCNN<\/a>. While\u00a0Object Detection<\/a> with Fast-RCNN ended up being the best fit, during our investigation we determined that each of these approaches involved different levels of complexity, each with its own strengths and weaknesses.<\/p>\n

Custom Vision Service<\/h3>\n

Training and consuming a REST-based service is dramatically easier than training, deploying and updating a custom computer vision model. As a result, we first investigated Microsoft’s\u00a0Custom Vision Service<\/a>. The Custom Vision Service is a tool for building custom image classifiers and improving them over time. \u00a0We trained a model on a sample data set of 882 images (sorted into 505 valid and 377 invalid images) containing standalone shelves of chocolate.<\/p>\n

\"Image<\/p>\n

We were able to train a relatively strong baseline model using the Custom Vision Service with the following benchmarks below:<\/p>\n

\"Image<\/p>\n

Additionally, we ran the model on an unseen dataset of 500 images to supplement this data and make sure that we had a consistent baseline.<\/p>\n

For more information on how to benchmark and put a Custom Vision Service Model into production, please see our previous code story, Food Classification with Custom Vision Service<\/a>. For a detailed explanation of standard classification metrics, please see\u00a0Metrics To Evaluate Machine Learning Algorithms in Python<\/a>.<\/div>\n
\n\n\n\n\n\n\n
\u00a0Label<\/strong><\/td>\nPrecision\u00a0<\/strong><\/td>\n\u00a0Recall<\/strong><\/td>\n\u00a0F-1 Score<\/strong><\/td>\nSupport<\/strong><\/td>\n<\/tr>\n
\u00a0Invalid<\/td>\n0.71<\/td>\n0.74<\/td>\n0.72<\/td>\n170<\/td>\n<\/tr>\n
Valid<\/td>\n0.87<\/td>\n0.85<\/td>\n0.86<\/td>\n353<\/td>\n<\/tr>\n
Avg \/ Total<\/td>\n0.82<\/td>\n0.81<\/td>\n0.82<\/td>\n523<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
Confusion matrix<\/strong><\/div>\n
<\/div>\n\n\n\n\n
125<\/td>\n45<\/td>\n<\/tr>\n
52<\/td>\n301<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
While the Custom Vision Service performed well for the given scenario and proved itself to be a very powerful image classification tool, the service had some limitations that made it prohibitive for a production use case.<\/div>\n

These limitations are best expressed in the following quote from the official\u00a0Custom Vision Service documentation<\/a>:<\/p>\n

The methods Custom Vision Service uses are robust to differences, which allows you to start prototyping with so little data. In theory v<\/span>ery few images are required to create a classifier — 30 images per class is enough to start your prototype. However, this means Custom Vision Service is usually not well suited to scenarios where you want to detect very subtle differences.<\/p><\/blockquote>\n

The\u00a0Custom Vision Service worked well when we narrowed down the scope of our policy problem to one policy and standalone shelves of chocolate, however, the service’s 1000 image training limit narrowed our ability to fine-tune the model around certain consistent but subtle policy edge cases.<\/p>\n

For example, the Custom Vision Service excelled at detecting the flagrant policy violations that represented the majority of our dataset, like the examples below:<\/p>\n

\"\"\u00a0 \u00a0\"\"<\/p>\n

However, it consistently failed to recognize more subtle yet persistent violations that were off by one chocolate, such as the first shelf in this picture:<\/p>\n

\"Image<\/p>\n

To transcend the limitations of the Custom Vision Service, we toyed with the idea of creating multiple models and then ensembling their results using a voting classifier<\/a>. While this would have no doubt improved the results of the model, and may be worth investigating for other scenarios, it would have also increased the API cost, as well as run-time. In addition, it would still not be scalable beyond one or two policies, since the\u00a0Custom Vision Service caps the number of models per account at nineteen.<\/p>\n

Transfer Learning using CNTK and ResNet<\/strong><\/h3>\n

To work around the dataset limits of the Custom Vision Service, we next investigated building an image recognition model with CNTK and Transfer learning<\/a> on top of ResNet with the following tutorial<\/a>. ResNet <\/a>is a deep convolutional neural network architecture developed by Microsoft for the image-net competition in 2015.<\/p>\n

In this evaluation, our training dataset contained two sets of 795 images representing valid and invalid policy.<\/p>\n

\"\"
Fig. 3: Representation of a ResNet CNN with an image from ImageNet. The input is an RGB image of a cat, the output is a probability vector, whose maximum corresponds to the label \u201ctabby cat\u201d.<\/figcaption><\/figure><\/p>\n

Since we did not have enough data (i.e., tens of thousands of samples) or the processing power to train our own large-scale CNN\u00a0model from scratch, we decided to leverage ResNet by retraining its output layer on our train dataset.<\/p>\n

Results<\/strong><\/h4>\n

We\u00a0ran the transfer learning ResNet model on three runs of 20, 200, and 2000 epochs respectively. We received the best results on our test dataset during our run of 2000 epochs.<\/p>\n\n\n\n\n\n\n
\u00a0Label<\/strong><\/td>\nPrecision\u00a0<\/strong><\/td>\n\u00a0Recall<\/strong><\/td>\n\u00a0F-1 Score<\/strong><\/td>\nSupport<\/strong><\/td>\n<\/tr>\n
\u00a0Invalid<\/td>\n0.38<\/td>\n0.96<\/td>\n0.54<\/td>\n171<\/td>\n<\/tr>\n
Valid<\/td>\n0.93<\/td>\n0.23<\/td>\n0.37<\/td>\n353<\/td>\n<\/tr>\n
Avg \/ Total<\/td>\n0.75<\/td>\n0.47<\/td>\n0.43<\/td>\n524<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Confusion Matrix<\/p>\n\n\n\n\n
165<\/td>\n6<\/td>\n<\/tr>\n
272<\/td>\n81<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

As we can see, the transfer learning approach severely under performed compared to the Custom Vision Service.<\/p>\n

Transfer learning on ResNet can be a powerful tool\u00a0for training strong object recognition with limited datasets. However, when we apply the model to new images that stray too far from the domain of the original 1000 ImageNet classes, the model struggles to capture new representative features due to its reused abstract features “learned” from the ImageNet<\/a> training set.<\/p>\n

Investigation Conclusions<\/strong><\/h4>\n

We saw promising results classifying individual policies using object recognition approaches, such as the Custom Vision Service considering the large quantity of brands and potential ways to order them on a shelf, it was prohibitive to determine policy adherence from an image using standard object recognition with the amount of available data.<\/p>\n

The complexity of the validation corner cases\u00a0combined with SMART Business’ concern about the ease of building new models for each policy using standard object recognition methods encouraged us to examine a more creative solution for their problem domain.<\/p>\n

\n

The Solution<\/h2>\n

Object Detection and Fast R<\/strong>–<\/strong>CNN<\/strong><\/h3>\n

To strengthen the policy signal while maintaining classification accuracy, we decided to use Object Detection and Fast R-CNN <\/a>with\u00a0AlexNet <\/a>to detect valid shelves in images. If we detected all valid shelves in a picture, then we could consider that stand as valid. In this way, we were not only able to classify our images but also to reuse pre-classified shelves to generate new configurable policies. We decided to use Fast R-CNN over alternatives such as Faster R-CNN since the implementation and evaluation pipeline was already proven to work well on top of CNTK see Object Detection Using CNTK<\/a>.<\/p>\n

First, we used the new image support feature of the Visual Object Tagging Tool (VoTT)<\/a> to tag a valid policy on a slightly larger 2600\u00a0image dataset. For an explanation of how to tag image directories with VOTT, see\u00a0Tagging an Image Directory<\/a>.<\/p>\n

\"\"
Note all three shelves are validly stocked, therefore this image represents a valid policy.<\/figcaption><\/figure><\/p>\n

By tweaking the filtration aspect ratios, quantity and minimum sizes of our region of interest, we were able to get quality results from our dataset.<\/p>\n<\/div>\n

Results<\/h2>\n

\"Image<\/p>\n

While the results of this model at first glance seem to be marginally worse than the results of the Custom Vision Service solution, the gains\u00a0in modularity\u00a0and the ability to generalize across certain persistent edge cases encouraged SMART Business to continue with the more advanced object detection approach.<\/p>\n

 <\/p>\n

\n

Opportunities for Reuse<\/h2>\n
\n

Below is an outline\u00a0of the advantages and disadvantages of the contextual methodologies for image classification that we investigated, in order of complexity.<\/p>\n<\/div>\n

\n\n\n\n\n\n\n
Methodology<\/b><\/td>\nAdvantages<\/b><\/td>\nDisadvantages <\/b><\/td>\nWhen to use<\/b><\/td>\n<\/tr>\n
Custom Vision Service<\/td>\n\n
    \n
  • Easy to get started from small datasets. No GPU required.<\/li>\n
  • Evaluated images can be retagged to improve model<\/li>\n
  • Supported service with one click to production.<\/li>\n<\/ul>\n<\/td>\n
\n
    \n
  • Struggles to detect subtle changes<\/li>\n
  • Cannot run model locally<\/li>\n
  • Limit of 1000 training images<\/li>\n<\/ul>\n<\/td>\n
\n
    \n
  • Cloud services such as the Custom Vision Service are well suited for Object Classification Problems where there is limited training data. This is the simplest methodology to attempt.<\/li>\n
  • In our investigations, the service performed the best on our dataset but struggled to scale to multiple policies and detect persistent edge cases.<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
CNN \/ Transfer Learning<\/td>\n\n
    \n
  • Leverages existing model layers so that model doesn\u2019t need to be trained from scratch.<\/li>\n
  • Easy to train, just need to point train script to sorted image directories.<\/li>\n
  • No training set size limit and models can run offline<\/li>\n<\/ul>\n<\/td>\n
\n
    \n
  • Struggles to classify data whose abstract features are dissimilar to those learned from Image-Net<\/li>\n
  • Needs GPU to train.<\/li>\n
  • More complex to put in production than the computer vision service.<\/li>\n<\/ul>\n<\/td>\n
\n
    \n
  • CNN Transfer learning on pre-trained models such as ResNet or Inception works best with mid-size datasets that share similar properties with ImageNet categories. Note if you have a large dataset with at least tens of thousands of samples it may be worthwhile retraining all the layers in a model.<\/li>\n
  • Of the methodologies we investigated transfer learning performed the worst for our complex classification scenario.<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n
Object Detection using VoTT<\/td>\n\n
    \n
  • Better suited for detecting subtle differences between image classes.<\/li>\n
  • Detected regions are modular and can be reused if complex classification criteria changes.<\/li>\n
  • No training set size limit and models can run offline<\/li>\n<\/ul>\n<\/td>\n
\n
    \n
  • Requires annotating bounding boxes on all images (though this is made easier with the VoTT tool).<\/li>\n
  • Needs GPU to train<\/li>\n
  • More complex to put in production than the computer vision service.<\/li>\n
  • Algorithms such as FastRCNN struggle to detect small areas.<\/li>\n<\/ul>\n<\/td>\n
\n
    \n
  • Using a combination of object detection and heuristics for image classification is well suited for scenarios where users have a midsized dataset yet need to detect subtle differences to differentiate image classes.<\/li>\n
  • Of the methodologies outlined this was the most complex to implement but provided the most robust results across our test set. This is the approach that SMART Business\u00a0decided to use.<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    The Deep Learning ecosystem is rapidly advancing with new algorithms being quickly iterated upon and released daily. After reading about state of the art performance on traditional benchmarks, it can be tempting to throw the newest available DNN algorithm at a classification problem. However, it is equally (if not more) important to evaluate these new technologies within the context of their application.\u00a0Too\u00a0often in machine learning, the use\u00a0of “new algorithms” overshadows the importance of well thought out\u00a0methodology.<\/p>\n<\/div>\n

    The methodologies investigated during our collaboration revealed a spectrum of classification methodologies of varying complexity and with trade-offs that are useful to consider when building image classification systems.<\/p>\n

    Our investigations show that it is important to\u00a0weigh the trade-offs of factors such as implementation\u00a0complexity, scalability, and optimization potential against others like dataset size, class instance variation, class similarity, and performance.<\/p>\n<\/div>\n

    <\/div>\n
    <\/div>\n","protected":false},"excerpt":{"rendered":"

    An overview of different image classification approaches including Microsoft Azure Custom Vision Service and CNTK for various levels of classification complexity.<\/p>\n","protected":false},"author":21353,"featured_media":10897,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[19],"tags":[123,279],"class_list":["post-3726","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-cntk","tag-object-detection"],"acf":[],"blog_post_summary":"

    An overview of different image classification approaches including Microsoft Azure Custom Vision Service and CNTK for various levels of classification complexity.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/3726","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21353"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=3726"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/3726\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/10897"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=3726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=3726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=3726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}