{"id":3057,"date":"2023-02-02T14:39:58","date_gmt":"2023-02-02T22:39:58","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/surface-duo\/?p=3057"},"modified":"2023-12-20T10:11:24","modified_gmt":"2023-12-20T18:11:24","slug":"onnx-machine-learning-1","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/surface-duo\/onnx-machine-learning-1\/","title":{"rendered":"On-device machine learning with ONNX"},"content":{"rendered":"<p>\n  Hello Android developers,\n<\/p>\n<p>\n  This week we\u2019re going to get started with on-device machine learning using the ONNX Runtime and check out an Android sample that identifies the objects using the camera video stream. \n<\/p>\n<h2>What is ONNX?<\/h2>\n<p>\n  ONNX stands for <em>Open Neural Network eXchange<\/em> and is an open-source format for AI models. ONNX supports interoperability between frameworks and optimization and acceleration options on each supported platform.\n<\/p>\n<p>\n  The <a href=\"https:\/\/onnxruntime.ai\/\">ONNX Runtime<\/a> is available across a large variety of platforms, and provides developers with the tools to run machine learning models locally. Pre-trained models can be <a href=\"https:\/\/onnxruntime.ai\/docs\/tutorials\/mobile\/helpers\/\">exported<\/a> to ONNX for distribution on a variety of platforms including Android. The <a href=\"https:\/\/onnxruntime.ai\/community.html\">ONNX community<\/a> includes many companies embedding in their products, providing models, and contributing to the ecosystem.\n<\/p>\n<h2>Get started with machine Learning on Android<\/h2>\n<p>\n  To see a machine learning model in action, follow the instructions for the <a href=\"https:\/\/onnxruntime.ai\/docs\/tutorials\/mobile\/deploy-android.html\">mobile image recognition example on Android<\/a> on the ONNX Runtime website. The example uses an existing pre-trained model \u2013 <a href=\"https:\/\/pytorch.org\/hub\/pytorch_vision_mobilenet_v2\/\">MOBILENET v2<\/a> \u2013 which is exported from ONNX format for use in the app. While there are <a href=\"https:\/\/github.com\/microsoft\/onnxruntime-inference-examples\/blob\/main\/quantization\/notebooks\/imagenet_v2\/mobilenet.ipynb\">instructions provided<\/a> to do this yourself it\u2019s easier to <a href=\"https:\/\/onnxruntimeexamplesdata.z13.web.core.windows.net\/mobilenet_v2_ort_models.zip\">download<\/a> them directly if you just want to see the sample run.\n<\/p>\n<p>\n  Once you have the ONNX model and classification files, clone the <a href=\"https:\/\/github.com\/microsoft\/onnxruntime-inference-examples\/tree\/main\/mobile\/examples\/image_classification\/android\">Android source code<\/a> from GitHub.\n<\/p>\n<p>\n  Before you build, copy the ONNX files to the <code>raw<\/code> resources directory and ensure the filenames match the Java resources references (<strong>MainActivity.java<\/strong> lines 144-154):\n<\/p>\n<pre>\/\/ Read MobileNet V2 classification labels\r\nprivate fun readLabels(): List&lt;String&gt; {\r\n    return resources.openRawResource(R.raw.imagenet_classes).bufferedReader().readLines()\r\n}\r\n\/\/ Read ort model into a ByteArray, run in background\r\nprivate suspend fun readModel(): ByteArray = withContext(Dispatchers.IO) {\r\n    val modelID =\r\n        if (enableQuantizedModel) R.raw.mobilenetv2_uint8 else R.raw.mobilenetv2_float\r\n    resources.openRawResource(modelID).readBytes()\r\n}<\/pre>\n<p>\n  Note that the <a href=\"https:\/\/onnxruntimeexamplesdata.z13.web.core.windows.net\/mobilenet_v2_ort_models.zip\">download<\/a> includes an older version <em>and<\/em> a v5 version of the models \u2013 I found that I needed the v5 files (which I renamed to match the resource names in code).\n<\/p>\n<p>\n  When you run the sample and grant camera permission, the model will display the top three best matches for the subject in the video frame:\n<\/p>\n<p>\n  <img decoding=\"async\" width=\"360\" height=\"773\" src=\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/02\/screenshot-showing-an-example-classification-of-a.jpeg\" class=\"wp-image-3058\" alt=\"Screenshot showing an example classification of a toy terrier dog\" srcset=\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/02\/screenshot-showing-an-example-classification-of-a.jpeg 360w, https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/02\/screenshot-showing-an-example-classification-of-a-140x300.jpeg 140w\" sizes=\"(max-width: 360px) 100vw, 360px\" \/>\n<\/p>\n<p><em>Figure 2: Inference percentages when detecting a dog breed<\/em>\n<\/p>\n<p>\n  This ONNX Android sample only performs image classification, but other mobile samples also cover basic usage, speech recognition, object detection, and vision-related models. Using these samples as a foundation you can incorporate other <a href=\"https:\/\/github.com\/onnx\/models\">trained models<\/a> into your apps!\n<\/p>\n<h2>Feedback and resources<\/h2>\n<p>\n  More information about the ONNX Runtime is available at <a href=\"https:\/\/onnxruntime.ai\/\">onnxruntime.ai<\/a>  and also on <a href=\"https:\/\/www.youtube.com\/onnxruntime\">YouTube<\/a>.\n<\/p>\n<p>\n  If you have any questions about applying machine learning, or would like to tell us about your apps, use the\u00a0<a href=\"http:\/\/aka.ms\/SurfaceDuoSDK-Feedback\" target=\"_blank\" rel=\"noopener\">feedback forum<\/a>\u00a0or message us on\u00a0<a href=\"https:\/\/twitter.com\/surfaceduodev\" target=\"_blank\" rel=\"noopener\">Twitter @surfaceduodev<\/a>.\n<\/p>\n<p>\n  There won\u2019t be a livestream this week, but check out the\u00a0<a href=\"https:\/\/youtube.com\/c\/surfaceduodev\" target=\"_blank\" rel=\"noopener\">archives on YouTube<\/a>. We\u2019ll see you online again soon!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hello Android developers, This week we\u2019re going to get started with on-device machine learning using the ONNX Runtime and check out an Android sample that identifies the objects using the camera video stream. What is ONNX? ONNX stands for Open Neural Network eXchange and is an open-source format for AI models. ONNX supports interoperability between [&hellip;]<\/p>\n","protected":false},"author":570,"featured_media":67,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[740],"tags":[729,728],"class_list":["post-3057","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-machine-learning","tag-onnx"],"acf":[],"blog_post_summary":"<p>Hello Android developers, This week we\u2019re going to get started with on-device machine learning using the ONNX Runtime and check out an Android sample that identifies the objects using the camera video stream. What is ONNX? ONNX stands for Open Neural Network eXchange and is an open-source format for AI models. ONNX supports interoperability between [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/3057","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/users\/570"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/comments?post=3057"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/3057\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media\/67"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media?parent=3057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/categories?post=3057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/tags?post=3057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}