{"id":5550,"date":"2017-12-05T14:32:27","date_gmt":"2017-12-05T22:32:27","guid":{"rendered":"\/developerblog\/?p=5550"},"modified":"2020-03-20T09:20:19","modified_gmt":"2020-03-20T16:20:19","slug":"comparing-transfer-learning-systems-custom-vision-service-vs-inception-vs-mobilenet","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/comparing-transfer-learning-systems-custom-vision-service-vs-inception-vs-mobilenet\/","title":{"rendered":"Comparing Image-Classification Systems: Custom Vision Service vs. Inception"},"content":{"rendered":"

We recently worked with Reverb<\/a>, an online marketplace for music gear. Musicians and music retailers use Reverb’s platform to buy and sell items such as guitars, keyboards, effects pedals, etc. A key element that aids discovery of items listed on the Reverb site is accurate metadata. For example, the manufacturer’s official name for the finish of a guitar\u00a0(e.g., “honey burst” or “fiesta red”) is an important criterion for surfacing relevant guitars in user searches. However, the finish name is sometimes omitted or not accurately entered when sellers list a guitar for sale.\u00a0To solve the problem of missing or mislabeled manufacturer finish metadata, we developed a set of image classification models with Reverb that help resolve manufacturer finish names based on product images.<\/p>\n

In this code story, we will cover how to build image classification models in Python with Custom Vision Service<\/a> and compare the results with popular Tensorflow-based models in terms of accuracy, prediction speed, training speed and setup complexity.<\/p>\n

Getting started with the Custom Vision Service<\/h2>\n

Custom Vision Service<\/a> is a managed service that enables developers to easily customize state-of-the-art deep learning image classification models using transfer learning. You can read more about the service in this code story<\/a>. We decided to use this service for our solution to classify the manufacturer finish names because it provides a very quick getting-started experience and enables us to build image classification models without having to manage and scale deep learning infrastructure.<\/p>\n

A simple way to train models via the Custom Vision Service is to use its web portal<\/a> and create classifiers via a drag-and-drop user interface to upload and tag training images. However, this approach is inefficient when dealing with large amounts of training data and when iterating on new models and ideas since the process cannot be automated. As a result, we created a Python SDK<\/a> for the Custom Vision service. Using this SDK, it is straightforward to quickly train new models with large amounts of training data.<\/p>\n

To get started, install the Custom Vision Python SDK and its dependencies:<\/p>\n

pip install custom_vision_client<\/pre>\n
We will assume your image data is laid out on disk using directories as labels like so:<\/p>\n
training-data\r\n\u251c\u2500\u2500 label1\r\n\u2502   \u251c\u2500\u2500 image1.jpg\r\n\u2502   \u2514\u2500\u2500 image2.jpg\r\n\u2514\u2500\u2500 label2\r\n    \u251c\u2500\u2500 image3.jpg\r\n    \u2514\u2500\u2500 image4.jpg\r\n<\/pre>\n
Now we\u2019re ready to build a model:<\/p>\n
from glob import glob\r\nfrom os.path import join\r\n\r\nfrom custom_vision_client import TrainingClient, TrainingConfig\r\n\r\nazure_region = \"southcentralus\"\r\ntraining_key = \"the-training-key\"  # from settings pane on customvision.ai\r\nproject_name = \"my-classifier\"\r\ndata_directory = \"\/path\/to\/training-data\"\r\n\r\ntraining_client = TrainingClient(TrainingConfig(azure_region, training_key))\r\n\r\n# create a new project, if the name already exists, a suffix will\r\n# be added to the name in order to make it unique\r\nproject_id = training_client.create_project(project_name).Id\r\n\r\nfor label in glob(data_directory):\r\n    images = glob(join(data_directory, label, '*.jpg'))\r\n    # register a new label if it doesn't already exist in the project\r\n    # also register training images, duplicates get ignored automatically\r\n    training_client.create_tag(project_id, label)\r\n    training_client.add_training_images(project_id, images, label)\r\n\r\nmodel_id = training_client.trigger_training(project_id).Id\r\n\r\nprint('{},{}'.format(project_id, model_id))  # record these for prediction\r\n<\/pre>\n
With Reverb, our objective was to classify an image into several categories of manufacturer finish names. These finish names roughly corresponded to colors and as such can be grouped naturally into distinct super-families like \u201cthe reds,\u201d \u201cthe greens,\u201d \u201cthe multi-colors\/sunburst,\u201d etc. We decided to utilize this natural structure of our data and build a hierarchical image classification model. First, we built a model to distinguish between the color super-families. Next, within each family, we built a second model to differentiate between the more detailed nuances of the finishes in that color family.<\/p>\n
The figure below shows a flow-chart diagram of the model:<\/p>\n
\"Image
Hierarchical model for classifying color finish names for guitars<\/figcaption><\/figure><\/p>\n
First, we predicted high-level color families such as red, green or sunburst. Within each family we then trained an additional model to differentiate nuances; for example, to tell the difference between “bordeaux-metallic” and “fiesta-red” within the “red” family.<\/p>\n
The intention behind this model was to maximize the amount of training data available to the first classifier and enable the second level of classifiers to learn more discriminatory features within each color family instead of having a single model learning features that span the entire label space. For the Reverb data set, this approach increased precision to 87% and recall to 84% compared to a standard multi-label model with precision at 79% and recall at 75%. The downside of the tiered approach is that we now require two classifications per image at prediction time which doubles latency. A single prediction takes on the order of 200ms-400ms so the latency for the hierarchical model is on the order of 400ms-800ms.<\/p>\n
\"Image
Chart showing the precision and recall of the Custom Vision Service as the amount of training data available to the model increases. We see that the service does well even with relatively limited amounts of training data.<\/figcaption><\/figure><\/p>\n
In synthesis, getting started building image classification models with the Custom Vision Service is very simple. Uploading images and building models can be done in a matter of minutes. No deep learning hardware needs to be set up or managed to train models. The Custom Vision Service leverages many data sets under the hood for transfer learning, so the performance of the trained models is strong, even with limited training data (see chart above). At prediction time, the latency for a request to the Custom Vision Service is on the order of 200ms to 400ms which is reasonable for many non-real-time applications (750ms to 1400ms if not in the same Azure data center). Note that the Custom Vision Service recently added support for exporting models<\/a> to enable prediction to be run locally on edge devices, so this latency can be further reduced.<\/p>\n
Getting started with Inception V3<\/h2>\n
In order to get an idea how Custom Vision Service compares in both prediction accuracy and speed to other popular convolutional networks, we built a pair of transfer learning implementations based on Inception V3 and MobileNet. For simplicity, the examples below are based on the command line tools included in TensorFlow’s examples.<\/p>\n
Both Inception V3 and MobileNet networks were retrained using the tensorflow\/tensorflow:1.3.0-devel-py3 Docker image<\/a>.\u00a0All experiments were done in a CPU-centric environment to mirror a production service deployment environment (that is, without Nvidia-Docker or GPU support due to the cost of running these in production at scale). From a working directory which includes the training images, you can start by running the following Docker container:<\/p>\n
docker run --rm -it -v `pwd`:\/c13n -w \/c13n tensorflow\/tensorflow:1.3.0-devel-py3 bash<\/pre>\n
Assuming the training images are laid out as described above, training can be done using the retrain.py<\/a> script like so:<\/p>\n
python3 \/tensorflow\/tensorflow\/examples\/image_retraining\/retrain.py \\\r\n    --architecture inception_v3 \\\r\n    --image_dir \/c13n\/training-data \\\r\n    --output_graph \/c13n\/index.pb \\\r\n    --output_labels \/c13n\/index.txt<\/pre>\n
After a few minutes, the script should complete with output similar to the following:<\/p>\n
INFO:tensorflow: Step 3999: Train accuracy = 79.0%\r\nINFO:tensorflow: Step 3999: Cross entropy = 1.548623\r\nINFO:tensorflow: Step 3999: Validation accuracy = 61.0% (N=100)\r\nINFO:tensorflow:Final test accuracy = 65.6% (N=570)\r\n<\/pre>\n
As you can see from the example, at 61.0% accuracy, the performance of Inception V3 trained on 5000 images is comparable to that of the Custom Vision Service trained on 200 images. Retraining of the Inception V3 neural network can take somewhere between 6-15 minutes per model whereas the Custom Vision Service only takes between 10 seconds to 5 minutes to train depending on the dataset size (timed on a 2.9 GHz Intel Core i7 machine with 16GB of RAM).<\/p>\n
In order to run a prediction using this newly-trained model we can run\u00a0the following command from TensorFlow’s examples:<\/p>\n
python3 \/tensorflow\/tensorflow\/examples\/label_image\/label_image.py \\\r\n    --image \/path\/to\/image.jpg \\\r\n    --input_width 128 \\\r\n    --input_height 128 \\\r\n    --graph \/c13n\/red.pb \\\r\n    --labels \/c13n\/red.txt \\\r\n    --input_layer DecodeJpeg \\\r\n    --output_layer final_result<\/pre>\n
Running a prediction using a local Inception V3 and a local image takes significantly longer (6000ms to 7000ms) than running the equivalent request on Custom Vision’s REST endpoint (750ms to 1400ms). We can shave about 1000ms off the prediction time by wrapping the Inception prediction code in a REST service (e.g. using hug<\/a>, shown in this\u00a0sample code<\/a>) and only loading the Inception model once at service start instead of for every prediction.<\/p>\n
Getting started with MobileNet<\/h2>\n
The steps for retraining and using a MobileNet model are almost identical as the those for Inception V3. The only difference is the value passed through the –architecture<\/span>\u00a0 command line parameter:<\/p>\n
python3 \/tensorflow\/tensorflow\/examples\/image_retraining\/retrain.py \\\r\n    --architecture mobilenet_0.25_128_quantized \\\r\n    --image_dir \/c13n\/training-data \\\r\n    --output_graph \/c13n\/index.pb \\\r\n    --output_labels \/c13n\/index.txt<\/pre>\n
Interestingly, running the training on the same set of 5,500 training images of size 128×128, the accuracy of the MobileNet model increases to 72.5%, higher than the one for Inception V3 (likely with some overfitting in MobileNet as hinted-at by the very high training accuracy).<\/p>\n
INFO:tensorflow: Step 3999: Train accuracy = 99.0%\r\nINFO:tensorflow: Step 3999: Cross entropy = 0.112992\r\nINFO:tensorflow: Step 3999: Validation accuracy = 75.0% (N=100)\r\nINFO:tensorflow:Final test accuracy = 72.5% (N=570)<\/pre>\n
However,\u00a0based on the community at large’s empirical experience, it’s likely that the performance of a MobileNet model on other data sets will be lower relative to Inception V3 since MobileNet is optimized for speed whereas Inception aims for correctness.<\/p>\n
Another noteworthy difference between Inception and MobileNet is the big savings in model size at 900KB for MobileNet vs 84MB for Inception V3. You can experiment further by switching between variants of MobileNet. For instance, using mobilenet_1.0_128<\/span>\u00a0 as the base model increases the model size to 17MB but also increases accuracy to 80.9%.<\/p>\n
INFO:tensorflow: Step 3999: Train accuracy = 100.0%\r\nINFO:tensorflow: Step 3999: Cross entropy = 0.034551\r\nINFO:tensorflow: Step 3999: Validation accuracy = 87.0% (N=100)\r\nINFO:tensorflow:Final test accuracy = 80.9% (N=570)\r\n<\/pre>\n
It’s also worth mentioning that this smaller model size is accompanied with a shorter prediction runtime of 950ms versus one of roughly 5000ms with Inception V3. If we wrap the MobileNet model in a REST service and load model only once during startup, one can achieve prediction speeds of around 400ms.<\/p>\n
For running prediction, the\u00a0label_image.py<\/a> script can be invoked for a MobileNet model as shown below. (Note that the input_layer and output_layer args are different than those used for Inception V3.)<\/p>\n
time python3 \/tensorflow\/tensorflow\/examples\/label_image\/label_image.py \\\r\n    --image \/path\/to\/image.jpg \\\r\n    --input_width 128 \\\r\n    --input_height 128 \\\r\n    --graph \/c13n\/red.mobilenet.pb \\\r\n    --labels \/c13n\/red.mobilenet.txt \\\r\n    --input_layer input \\\r\n    --output_layer final_result\r\n\r\nbordeaux metallic 0.999991\r\nburgundy mist metallic 8.92877e-06\r\nburgundy mist 1.87918e-07\r\ndakota red 4.31156e-09\r\n\r\nreal 0m0.955s\r\nuser 0m0.840s\r\nsys 0m0.190s\r\n<\/pre>\n
Framework comparison<\/h2>\n
The figures below show a comparison of the predictions of the Custom Vision Service, Inception and MobileNet models on a variety of sample guitar images with diverse rotations, backgrounds, lighting and cropping factors. Overall, MobileNet is the fastest model at prediction time and the Custom Vision Service is the most accurate while having an acceptable runtime.<\/p>\n
\"Image
Sample guitar images alongside their color finish name as predicted by the Tensorflow-based MobileNet and Inception models as well as the predictions made by the Custom Vision Service. We see that the Custom Vision Service does well across a wide range of guitar images, rotations, partial views, lighting changes, etc.<\/figcaption><\/figure><\/p>\n
\"Image
Charts comparing the Tensorflow-based models MobileNet and Inception with the Custom Vision Service in terms of accuracy, model size, training time and prediction time. We see that the Custom Vision Service has the best aggregate performance across the categories.<\/figcaption><\/figure><\/p>\n
Summary<\/h2>\n
In this code story, we’ve shown how to tackle a custom image classification task via transfer learning on the Custom Vision Service and Tensorflow. Overall, we found the Custom Vision Service to have the best performance on our task. Due to the high-end models backing the service, we expect the great performance can be generalized to other image classification tasks such as identifying sign language utterances in a video stream or classifying skin disease based on smartphone camera pictures.<\/p>\n
The Custom Vision Service offers a simple getting-started scenario enabling us to build models in minutes via a simple Python SDK<\/a> without having to set up a complicated Tensorflow stack. If you’re interested in trying out the Custom Vision Service to get started with your image classification project, take a look at the\u00a0Python SDK<\/a>\u00a0and get in touch<\/a> with us on GitHub! We’d love to hear your feedback.<\/p>\n
Resources<\/h2>\n