{"id":36632,"date":"2018-04-23T14:51:28","date_gmt":"2018-04-23T21:51:28","guid":{"rendered":"https:\/\/blog.xamarin.com\/?p=36632"},"modified":"2019-04-04T07:47:00","modified_gmt":"2019-04-04T14:47:00","slug":"android-apps-tensorflow","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/xamarin\/android-apps-tensorflow\/","title":{"rendered":"Using TensorFlow and Azure to Add Image Classification to Your Android Apps"},"content":{"rendered":"

\t\t\t\tTensorFlow<\/em> is a well established, open source machine learning and deep learning framework that can be used to create and run a wide range of different models, usually using powerful machines in the cloud. In addition, TensorFlow also supports running models on mobile devices through the TensorFlow.Mobile library<\/a>, taking advantage of the hardware acceleration available on modern phones to run models incredibly fast on low powered mobile devices.<\/p>\n

In this post, we’ll discuss how to build an image classifier to identify different fruits using the Azure Custom Vision service, and create a simple Android app to use this model to classify images.<\/p>\n

Getting Started<\/h2>\n

Creating ML models can be time consuming and require large data sets. To make it easier to create image classification models,\u00a0Microsoft has created the Custom Vision Service<\/a>, which uses a technique called transfer learning to allow you to train an image classifier using only a small number of images, instead of the thousands that traditionally would be required to train such a model. These models can be exported as TensorFlow models (or CoreML if you are on iOS<\/a>) and used from inside an Android app.<\/p>\n

Lets start by creating a project in the Custom Vision service. Make sure you select one of the compact<\/code> domains, the compact domains create models that are small enough to be exported and used from a mobile device.<\/p>\n

\"Creating<\/p>\n

Image classifiers are trained by being given images with one or more tags<\/strong>, which are text labels that indicate what is in that image. For example, you might upload five pictures of oranges with the tag of orange<\/code>, five pictures of bananas with the tag banana<\/code>,\u00a0and so on for different fruits.<\/p>\n

To train your model, you will need at least five images for each tag you want to use, so the first step is to gather the data by photographing whatever fruit you have to hand. Next, click the Add Images<\/strong>\u00a0button and upload the images of the first fruit, setting the appropriate tag. Repeat this for the rest of the fruits. Finally click the Train<\/strong> button on the top menu to train the model.<\/p>\n

\"Uploading<\/p>\n

Once trained, click the Export<\/strong> button to export the model for use with TensorFlow.<\/p>\n

\"Exporting<\/p>\n

This will download a zip file containing two files: model.pb<\/em> and labels.txt<\/em>.<\/p>\n

Adding the Model to Your Android App<\/h2>\n

Now that we have the model, it’s time to add it to an Android app project and use it to classify images.<\/p>\n

Add the Model and TensorFlow.mobile to the Project<\/h3>\n

Create a new Android app by going to File -> New Project -> Android -> Single View App<\/strong> in Visual Studio. Add the model.pb<\/em>\u00a0and labels.txt<\/em>\u00a0files to the Assets<\/code>\u00a0folder, and make sure the build action is set to Android Asset<\/code>.<\/p>\n

TensorFlow.Mobile is available as a NuGet package that wraps the native Android libraries. Install the Xam.Android.Tensorflow<\/a><\/strong> package in your Android app.<\/p>\n

Initializing the Model and Labels<\/h3>\n

To use the model, we need to create an instance of the TensorFlow interface using our model file. We also need to load the labels file; the model from TensorFlow doesn’t contain the actual tag values, so these come from the labels file.<\/p>\n

\n\n    using using Org.Tensorflow.Contrib.Android;\n    ...\n    var assets = Application.Context.Assets;\n    inferenceInterface = new TensorFlowInferenceInterface(assets, \"model.pb\");\n    var sr = new StreamReader(assets.Open(\"labels.txt\"));\n    var labels = sr.ReadToEnd()\n                   .Split('\\n')\n                   .Select(s => s.Trim())\n                   .Where(s => !string.IsNullOrEmpty(s))\n                   .ToList();\n<\/code>\n<\/pre>\n
Recognizing Images<\/h2>\n
To recognize an image, first we need to capture one from somewhere. To use the camera, I’d recommend using the Xam.Plugin.Media<\/strong> NuGet package<\/a>. This package has helpers to open the camera and extract the photo taken by the user.<\/p>\n
Preparing the Input for the Model<\/h3>\n
Once you have a photo, it needs to be converted for the Custom Vision model. These models require the input to be of the correct size and format to work; they don’t understand images, as such, but instead understand binary data. For the models created by the Custom Vision service, you need the images to be 227×227 pixels in size, converted to an array of floating point values, one per red, green, and blue value for each pixel, with some model-specific adjustments to the color value. The following code converts an Android Bitmap<\/code> to the correct format.<\/p>\n
\n\n    var resizedBitmap = Bitmap.CreateScaledBitmap(bitmap, 227, 227, false)\n                              .Copy(Bitmap.Config.Argb8888, false);\n    var floatValues = new float[227 * 227 * 3];\n    var intValues = new int[227 * 227];\n    resizedBitmap.GetPixels(intValues, 0, 227, 0, 0, 227, 227);\n    for (int i = 0; i > 8) & 0xFF) - 117);\n        floatValues[i * 3 + 2] = (((val >> 16) & 0xFF) - 123);\n    }\n<\/code>\n<\/pre>\n
Running the Model<\/h3>\n
Now that we have our image in the right format, we can run the model against our data. Running the model involves feeding it named inputs, running it for named outputs, then retrieving the output you’re interested in by name. Each model can take multiple inputs and give multiple outputs, but in our case we only have one input and need one output. The following code shows this for our Custom Vision model.<\/p>\n
\n\n    var outputs = new float[labels.Count];\n    inferenceInterface.Feed(\"Placeholder\", floatValues, 1, 227, 227, 3);\n    inferenceInterface.Run(new[] { \"loss\" });\n    inferenceInterface.Fetch(\"loss\", outputs);\n<\/code>\n<\/pre>\n
The outputs<\/code> array is filled with floating point values, each one representing the confidence that the image matching a tag in the labels list at the same array position. For example, if you had three labels in the list in the following order:<\/p>\n