Android and iOS provide built-in capabilities for on-device inferencing with technologies such as TensorFlow Lite<\/a> and Core ML<\/a>.<\/p>\n ONNX Runtime<\/a> has recently added support for Xamarin<\/a> and can be integrated into your mobile application to execute cross-platform on-device inferencing of ONNX (Open Neural Network Exchange)<\/a> models. It already powers machine learning models in key Microsoft products and services across Office, Azure, Bing, as well as other community projects. You can create your own ONNX models, using services such as Azure Custom Vision<\/a>, or convert existing models to ONNX format<\/a>.<\/p>\n This post walks through an example demonstrating the high-level steps for leveraging ONNX Runtime<\/a> in a Xamarin.Forms app for on-device inferencing. An existing open-source image classification model (MobileNet<\/a>) from the ONNX Model Zoo<\/a> has been used for this example.<\/p>\n Our sample Xamarin.Forms app<\/a> classifies the primary object in the provided image<\/a>, a golden retriever in this case, and displays the result with the highest score.<\/p>\n The following changes were made to the Blank App<\/strong><\/a> template which includes a common .NET Standard<\/a> project along with Android and iOS targets.<\/p>\n Two classes from the common project we’ll be focused on are:<\/p>\n MobileNetImageClassifier<\/strong> encapsulates the use of the model<\/a> via ONNX Runtime<\/a> as per the model documentation<\/a>. You can see visualizations of the model’s network architecture, including the expected names, types, and shapes (dimensions) for its inputs and outputs using Netron<\/a>.<\/p>\n This class exposes two public methods GetSampleImageAsync<\/strong> and GetClassificationAsync<\/strong>. The former loads a sample image for convenience and the latter performs the inferencing on the supplied image. Here’s a breakdown of the key steps.<\/p>\n Initialization involves loading those embedded resource files representing the model<\/a>, labels<\/a>, and sample image<\/a>. The asynchronous initialization pattern<\/a> is used to simplify its use downstream while preventing use of those resources before initialization work has completed. Our constructor starts the asynchronous initialization by calling InitAsync<\/strong>. The initialization Task is stored so several callers can await the completion of the same operation. In this case, the GetSampleImageAsync<\/strong> and GetClassificationAsync<\/strong> methods.<\/p>\n Raw images must be transformed according to the requirements of the model<\/a> so it matches how the model was trained. Image data is then stored in a contiguous sequential block of memory, represented by a Tensor<\/a> object, as input for inferencing.<\/p>\n Our first step in this process is to resize the original image if necessary so height and width is at least 224. In this case, images are first resized so the shortest edge is 224 then center-cropped so the longest edge is also 224. For the purposes of this example, SkiaSharp<\/a> has been used to handle the requisite image processing.<\/p>\n The second step is to normalize the resulting image pixels and store them in a flat array that can be used to create the Tensor<\/a> object. In this case, the model expects the R, G, and B values to be in the range of [0, 1] normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. The loop below iterates over the image pixels one row at a time, applies the requisite normalization to each value, then stores each in the channelData array. Our channelData array stores the normalized R,G, and B values sequentially; first all the R values, then all the G, then all the B (instead of the original sequence i.e. RGB, RGB, etc.<\/p>\n This channelData array is then used to create the requisite Tensor<\/a> object as input to the InferenceSession<\/a> Run method<\/a>. The dimensions (1, 3, 224, 224) represent the shape of the Tensor<\/a> required by the model. In this case, mini-batches of 3-channel RGB images that are expected to have a height and width of 224.<\/p>\n An InferenceSession<\/a> is the runtime representation of an ONNX model. It’s used to run the model with a given input returning the computed output values. Both the input and output values are collections of NamedOnnxValue<\/a> objects representing name-value pairs of string names and Tensor<\/a> objects.<\/p>\n This model outputs a score for each classification. Our code resolves the Tensor<\/a> by name and gets the highest score value for simplicity in this example. The corresponding label item is then used as the return value for the GetClassificationAsync<\/strong> method. Additional work would be required to calculate the softmax probability if you wanted to include an indication of confidence alongside the label.<\/p>\n Our MainPage<\/strong> XAML features a single Button<\/a> for running the inference via the MobileNetImageClassifier<\/strong>. It resolves the sample image then passes it into the GetClassificationAsync<\/strong> method before displaying the result via an alert.<\/p>\n This first-principles example demonstrates basic inferencing with ONNX Runtime<\/a> and leverages the default options for the most part. There are several optimizations recommended by the ONNX Runtime documentation<\/a> that can be particularly beneficial for mobile.<\/p>\n You can accelerate inference speed by reusing the same InferenceSession<\/a> across multiple inference runs to avoid unnecessary allocation\/disposal overhead.<\/p>\n Several examples create the Tensor<\/a> object first and then set values on it directly. This is simpler and easier to follow since it avoids the need to perform the offset calculations used in this example. However, it’s faster to prepare a primitive array first then use that to create the Tensor<\/a> object. The trade-off here is between simplicity and performance.<\/p>\n ONNX Runtime executes models using the CPU EP (Execution Provider<\/a>) by default. It’s possible to use the NNAPI EP<\/a> (Android) or the Core ML EP<\/a> (iOS) for ORT format models instead by using the appropriate SessionOptions<\/a> when creating an InferenceSession<\/a>. These may or may not offer better performance depending on how much of the model can be run using NNAPI<\/a> \/ Core ML<\/a> and the device capabilities. It’s worth testing with and without the platform-specific EPs then choosing what works best for your model. There are also several options per EP. See NNAPI Options<\/a> and Core ML Options<\/a> for more detail on how this can impact performance and accuracy.<\/p>\n The platform-specific EPs can be configured via the SessionOptions<\/a>. For example:<\/p>\n SessionOptionsContainer<\/strong> can be used to simplify use of platform-specific SessionOptions<\/a> from common platform-agnostic code. For example:<\/p>\n If you have access to the data that was used to train the model you can explore quantizing the model. At a high-level, quantization<\/a> in ONNX Runtime involves mapping higher precision floating point values to lower precision 8-bit values. This topic is covered in detail in the ONNX performance tuning documentation<\/a>, but the idea is to trade off some precision in order to reduce the size of the model and increase performance. It can be especially relevant for mobile apps where package size, battery life, and hardware constraints are key considerations. You can test this by switching out the mobilenetv2-7.onnx<\/a> model, used in this example, for the mobilenetv2-7-quantized.onnx<\/a> model included in the same repo. Quantization can only be performed on those models that use opset<\/a> 10 and above. However, models using older opsets can be updated using the VersionConverter<\/a> tool.<\/p>\n There are several existing ONNX models available such as those highlighted in the ONNX Model Zoo<\/a> collection. Not all of these are optimized for mobile but you’re not limited to using models already in ONNX format. You can convert existing pre-trained ready to deploy models from other popular sources and formats, such as PyTorch Hub<\/a>, TensorFlow Hub<\/a>, and SciKit-Learn<\/a>. The following resources provide guidance on how to convert each of these respective models to ONNX format:<\/p>\n Your pre-processing must of course match how the model was trained and so you’ll need to find the model-specific instructions for the model you convert.<\/p>\n In Visual Studio 2022<\/a>, Hot Reload<\/a> loads some additional dependencies including System.Memory<\/a> and System.Buffers<\/a> which may cause conflicts with packages such as ONNX Runtime<\/a>. You can Disable Hot Reload<\/a> as a workaround until the issue<\/a> has been addressed.<\/p>\n The intent of this post was to provide a helpful on-ramp for those looking to leverage ONNX Runtime<\/a> in their Xamarin.Forms apps. Be sure to checkout the official getting started<\/a> and tutorial<\/a> content as well as the Xamarin samples<\/a>.<\/p>\n Learn how to integrate machine learning into your Xamarin.Forms application with ONNX Runtime<\/p>\n","protected":false},"author":1967,"featured_media":49028,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[313,303,1,291,367],"tags":[3617,9214,9215,27,16],"class_list":["post-49014","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-android","category-ios","category-xamarin","category-xamarin-platform","category-xamarin-forms","tag-machine-learning","tag-onnx","tag-onnx-runtime","tag-xamarin","tag-xamarin-forms"],"acf":[],"blog_post_summary":" Learn how to integrate machine learning into your Xamarin.Forms application with ONNX Runtime<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/49014","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/users\/1967"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/comments?post=49014"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/49014\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/media\/49028"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/media?parent=49014"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/categories?post=49014"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/tags?post=49014"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Getting Started<\/h2>\n
![\"ONNX](\"https:\/\/devblogs.microsoft.com\/xamarin\/wp-content\/uploads\/sites\/44\/2021\/12\/inference_result.png\")
<\/p>\n\n
Inferencing With ONNX Runtime<\/h2>\n
\n
MobileNetImageClassifier<\/h3>\n
Initialization<\/h4>\n
const int DimBatchSize = 1;\nconst int DimNumberOfChannels = 3;\nconst int ImageSizeX = 224;\nconst int ImageSizeY = 224;\nconst string ModelInputName = \"input\";\nconst string ModelOutputName = \"output\";\n\nbyte[] _model;\nbyte[] _sampleImage;\nList<string> _labels;\nInferenceSession _session;\nTask _initTask;\n\n...\n\npublic MobileNetImageClassifier()\n{\n _ = InitAsync();\n}\n\nTask InitAsync()\n{\n if (_initTask == null || _initTask.IsFaulted)\n _initTask = InitTask();\n\n return _initTask;\n}\n\nasync Task InitTask()\n{\n var assembly = GetType().Assembly;\n\n \/\/ Get labels\n using var labelsStream = assembly.GetManifestResourceStream($\"{assembly.GetName().Name}.imagenet_classes.txt\");\n using var reader = new StreamReader(labelsStream);\n\n string text = await reader.ReadToEndAsync();\n _labels = text.Split(new string[] { Environment.NewLine }, StringSplitOptions.RemoveEmptyEntries).ToList();\n\n \/\/ Get model and create session\n using var modelStream = assembly.GetManifestResourceStream($\"{assembly.GetName().Name}.mobilenetv2-7.onnx\");\n using var modelMemoryStream = new MemoryStream();\n\n modelStream.CopyTo(modelMemoryStream);\n _model = modelMemoryStream.ToArray();\n _session = new InferenceSession(_model);\n\n \/\/ Get sample image\n using var sampleImageStream = assembly.GetManifestResourceStream($\"{assembly.GetName().Name}.dog.jpg\");\n using var sampleImageMemoryStream = new MemoryStream();\n\n sampleImageStream.CopyTo(sampleImageMemoryStream);\n _sampleImage = sampleImageMemoryStream.ToArray();\n}<\/code><\/pre>\nPreprocessing<\/h4>\n
using var sourceBitmap = SKBitmap.Decode(image);\nvar pixels = sourceBitmap.Bytes;\n\nif (sourceBitmap.Width != ImageSizeX || sourceBitmap.Height != ImageSizeY)\n{\n float ratio = (float)Math.Min(ImageSizeX, ImageSizeY) \/ Math.Min(sourceBitmap.Width, sourceBitmap.Height);\n\n using SKBitmap scaledBitmap = sourceBitmap.Resize(new SKImageInfo(\n (int)(ratio * sourceBitmap.Width), \n (int)(ratio * sourceBitmap.Height)), \n SKFilterQuality.Medium);\n\n var horizontalCrop = scaledBitmap.Width - ImageSizeX;\n var verticalCrop = scaledBitmap.Height - ImageSizeY;\n var leftOffset = horizontalCrop == 0 ? 0 : horizontalCrop \/ 2;\n var topOffset = verticalCrop == 0 ? 0 : verticalCrop \/ 2;\n\n var cropRect = SKRectI.Create(\n new SKPointI(leftOffset, topOffset), \n new SKSizeI(ImageSizeX, ImageSizeY));\n\n using SKImage currentImage = SKImage.FromBitmap(scaledBitmap);\n using SKImage croppedImage = currentImage.Subset(cropRect);\n using SKBitmap croppedBitmap = SKBitmap.FromImage(croppedImage);\n\n pixels = croppedBitmap.Bytes;\n}<\/code><\/pre>\nvar bytesPerPixel = sourceBitmap.BytesPerPixel;\nvar rowLength = ImageSizeX * bytesPerPixel;\nvar channelLength = ImageSizeX * ImageSizeY;\nvar channelData = new float[channelLength * 3];\nvar channelDataIndex = 0;\n\nfor (int y = 0; y < ImageSizeY; y++)\n{\n var rowOffset = y * rowLength;\n\n for (int x = 0, columnOffset = 0; x < ImageSizeX; x++, columnOffset += bytesPerPixel)\n {\n var pixelOffset = rowOffset + columnOffset;\n\n var pixelR = pixels[pixelOffset];\n var pixelG = pixels[pixelOffset + 1];\n var pixelB = pixels[pixelOffset + 2];\n\n var rChannelIndex = channelDataIndex;\n var gChannelIndex = channelDataIndex + channelLength;\n var bChannelIndex = channelDataIndex + (channelLength * 2);\n\n channelData[rChannelIndex] = (pixelR \/ 255f - 0.485f) \/ 0.229f;\n channelData[gChannelIndex] = (pixelG \/ 255f - 0.456f) \/ 0.224f;\n channelData[bChannelIndex] = (pixelB \/ 255f - 0.406f) \/ 0.225f;\n\n channelDataIndex++;\n }\n}<\/code><\/pre>\nvar input = new DenseTensor<float>(channelData, new[] \n{ \n DimBatchSize, \n DimNumberOfChannels, \n ImageSizeX, \n ImageSizeY \n});<\/code><\/pre>\nInferencing<\/h4>\n
using var results = _session.Run(new List<NamedOnnxValue> \n{ \n NamedOnnxValue.CreateFromTensor(ModelInputName, input) \n});<\/code><\/pre>\nPostprocessing<\/h4>\n
var output = results.FirstOrDefault(i => i.Name == ModelOutputName);\nvar scores = output.AsTensor<float>().ToList();\nvar highestScore = scores.Max();\nvar highestScoreIndex = scores.IndexOf(highestScore);\nvar label = _labels.ElementAt(highestScoreIndex);<\/code><\/pre>\nMainPage<\/h3>\n
var sampleImage = await _classifier.GetSampleImageAsync();\nvar result = await _classifier.GetClassificationAsync(sampleImage);\nawait DisplayAlert(\"Result\", result, \"OK\");<\/code><\/pre>\nOptimizations and Tips<\/h2>\n
Reuse InferenceSession objects<\/h3>\n
Consider whether to set values directly on an existing Tensor or create using an existing array<\/h3>\n
Experiment with different EPs (Execution Providers<\/a>)<\/h3>\n
Use NNAPI on Android<\/h4>\n
options.AppendExecutionProvider_Nnapi();<\/code><\/pre>\nUse Core ML on iOS<\/h4>\n
options.AppendExecutionProvider_CoreML(CoreMLFlags.COREML_FLAG_ONLY_ENABLE_DEVICE_WITH_ANE)<\/code><\/pre>\nRegister named platform-specific SessionOptions configuration<\/h4>\n
\/\/ Android\nSessionOptionsContainer.Register(\"sample_options\", (sessionOptions) \n => sessionOptions.AppendExecutionProvider_Nnapi());\n\n\/\/ iOS\nSessionOptionsContainer.Register(\"sample_options\", (sessionOptions) \n => sessionOptions.AppendExecutionProvider_CoreML(\n CoreMLFlags.COREML_FLAG_ONLY_ENABLE_DEVICE_WITH_ANE));\n<\/code><\/pre>\nUse in common platform-agnostic code<\/h4>\n
\/\/ Apply named configuration to new SessionOptions\nvar options = SessionOptionsContainer.Create(\"sample_options\");\n\n...\n\n\/\/ Alternatively, apply named configuration to existing SessionOptions\noptions.ApplyConfiguration(\"sample_options\");<\/code><\/pre>\nQuantize models to reduce size and execution time<\/h3>\n
There are several pre-trained ready to deploy models available<\/h3>\n
\n
Disable Hot Reload if you hit a MissingMethodException<\/a> related to ReadOnlySpan
Summary<\/h2>\n
Useful links<\/h2>\n
\n
![sample photo<\/a>,](\"https:\/\/github.com\/microsoft\/onnxruntime\/raw\/master\/csharp\/sample\/Microsoft.ML.OnnxRuntime.ResNet50v2Sample\/dog.jpeg\"){kind=link}