{"id":3079,"date":"2023-02-24T16:28:33","date_gmt":"2023-02-25T00:28:33","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/surface-duo\/?p=3079"},"modified":"2023-12-20T10:10:17","modified_gmt":"2023-12-20T18:10:17","slug":"onnx-machine-learning-4","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/surface-duo\/onnx-machine-learning-4\/","title":{"rendered":"Built-in model pre-processing with ONNX"},"content":{"rendered":"

\n Hello Android developers,\n<\/p>\n

\n Previously we looked at how to pre-process image inputs<\/a> for an ONNX model using Kotlin. It\u2019s useful to understand this process because the principles apply to any model that you wish to use. On the other hand, having to write boilerplate code for input processing can be tedious \u2013 it also means there\u2019s more code that could be buggy and require testing.\n<\/p>\n

\n The ONNXRuntime-Extensions<\/a> project simplifies this by including custom operators for common pre- and post-processing of model inputs. This means you can build, test, and deploy models that include the necessary input pre-processing and output post-processing, making it even easier to incorporate into application projects like Android.\n<\/p>\n

\n The super resolution sample<\/a> demonstrates how to add pre- and post-processing to a model and run it with just a few lines of code.\n<\/p>\n

Add pre-processing to the model<\/h2>\n

\n Instead of having to write the input image resizing and other manipulations in Kotlin, the pre- and post-processing steps can be added to the model using operators available in newer versions of the ONNX runtime (eg. ORT 1.14\/opset 18) or in the onnxruntime_extensions<\/code> library. \n<\/p>\n

\n The super-resolution sample is a great demonstration of this feature as both the input and output are images, so including the processing steps in the model greatly reduces platform-specific code.\n<\/p>\n

\n The steps to add the pre- and post-processing to the superresolution<\/code> model are in the Prepare the model<\/a> section of the sample documentation. The python script that creates the updated model can be viewed at superresolution_e2e.py<\/a>. \n<\/p>\n

The superresolution_e2e script is explained step-by-step on pytorch.org<\/a><\/p><\/blockquote>\n

\n When you run the script as instructed, it produces two models in ONNX format \u2013 the basic pytorch_superresolution.onnx<\/strong> model and another version that includes additional processing pytorch_superresolution_with_pre_and_post_proceessing.onnx<\/strong>. The second model, including the processing instructions, can be called in an Android app with fewer lines of code than the previous sample<\/a> we discussed.\n<\/p>\n

Comparing the models<\/h2>\n

\n Before we dive into the Android sample app, the difference between the two models can be observed using the online tool Netron.app<\/a>. Netron helps to visualize various neural network, deep learning and machine learning models.\n<\/p>\n

\n The first model \u2013 requiring native pre-processing and post-processing of the input and output \u2013 shows the following:\n<\/p>\n

\n \n<\/p>\n

Figure 1: superresolution model converted to ONNX and viewed in Netron<\/em>\n<\/p>\n

\n This is a relatively simple model, and it\u2019s not important to understand the graph for now, just to compare it to the model that includes the pre- and post-processing operations shown below:\n<\/p>\n

\n \n<\/p>\n

Figure 2: superresolution model with input and output processing, converted to ONNX and viewed in Netron<\/em>\n<\/p>\n

\n The additional operations for format and resize image bytes are chained before and after the original model. Once again diving into the details of the graph isn\u2019t important, it\u2019s included here to illustrate that additional operations have been packaged into the model to make it easier to consume with less native code on any supported platform.\n<\/p>\n

Integrating with Android<\/h2>\n

\n The Android sample that can host this model is available on GitHub<\/a>. The initialization step in MainActivity.kt<\/a> is similar to the image classifier sample<\/a> but with the addition of the sessionOptions<\/code> object where the ONNX runtime extensions are added to the session. Without the extensions, the model with extra processing might be missing operations required to function. \n<\/p>\n

\n This code snippet highlights the key lines for creating the ONNX runtime environment class:\n<\/p>\n

  var ortEnv: OrtEnvironment = OrtEnvironment.getEnvironment()\r\n  \/\/ fun onCreate\r\n  val sessionOptions: OrtSession.SessionOptions = OrtSession.SessionOptions()\r\n  sessionOptions.registerCustomOpLibrary(OrtxPackage.getLibraryPath())\r\n  ortSession = ortEnv.createSession(readModel(), sessionOptions) \/\/ the model is in raw resources\r\n  \/\/ fun performSuperResolution\r\n  var superResPerformer = SuperResPerformer()\r\n  var result = superResPerformer.upscale(readInputImage(), ortEnv, ortSession)\r\n  \/\/ result.outputBitmap contains the output image!<\/pre>\n
\n  The superResPerformer.upscale<\/code> function is shown below.\n<\/p>\n
\n  The code to run the model in SuperResPerformer.kt<\/a> has a lot less code than the image classifier sample, because there is no native processing required (like the ImageUtil.kt<\/a> helper class, which is no longer needed). Image bytes can be used to create a tensor and image bytes are emitted as the result. ortSession.run<\/code> is the key function takes the inputTensor<\/code> and returns the resulting upscaled image:\n<\/p>\n
fun upscale(inputStream: InputStream, ortEnv: OrtEnvironment, ortSession: OrtSession): Result {\r\n    var result = Result()\r\n    \/\/ Step 1: convert image into byte array (raw image bytes)\r\n    val rawImageBytes = inputStream.readBytes()\r\n    \/\/ Step 2: get the shape of the byte array and make ort tensor\r\n    val shape = longArrayOf(rawImageBytes.size.toLong())\r\n    val inputTensor = OnnxTensor.createTensor(\r\n        ortEnv,\r\n        ByteBuffer.wrap(rawImageBytes),\r\n        shape,\r\n        OnnxJavaType.UINT8\r\n    )\r\n    inputTensor.use {\r\n        \/\/ Step 3: call ort inferenceSession run\r\n        val output = ortSession.run(Collections.singletonMap(\"image\", inputTensor))\r\n        \/\/ Step 4: output analysis\r\n        output.use {\r\n            val rawOutput = (output?.get(0)?.value) as ByteArray\r\n            val outputImageBitmap =\r\n                byteArrayToBitmap(rawOutput)\r\n            \/\/ Step 5: set output result\r\n            result.outputBitmap = outputImageBitmap\r\n        }\r\n    }\r\n    return result\r\n}<\/pre>\n
\n  A screenshot from the Android super resolution demo<\/a> is shown below:\n<\/p>\n
\n  \"Super\n<\/p>\n
Figure 3: Android demo \u201csuperresolution\u201d that improves resolution of an image<\/em>\n<\/p>\n
\n  This particular upscaling model is relatively simple, but the concepts behind packaging models to make them easier to consume can be applied to other models with pre- and post-processing requirements.\n<\/p>\n
Resources and feedback<\/h2>\n
\n  More information about the ONNX Runtime is available at\u00a0onnxruntime.ai<\/a>\u00a0and also on\u00a0YouTube<\/a>. You can read more<\/a> about Netron.app as well as find the source and downloads<\/a> on GitHub.\n<\/p>\n
\n  If you have any questions about applying machine learning, or would like to tell us about your apps, use the\u00a0feedback forum<\/a>\u00a0or message us on\u00a0Twitter @surfaceduodev<\/a>.\n<\/p>\n
\n  There won\u2019t be a livestream this week, but check out the\u00a0archives on YouTube<\/a>. We\u2019ll see you online again soon!<\/p>\n","protected":false},"excerpt":{"rendered":"
Hello Android developers, Previously we looked at how to pre-process image inputs for an ONNX model using Kotlin. It\u2019s useful to understand this process because the principles apply to any model that you wish to use. On the other hand, having to write boilerplate code for input processing can be tedious \u2013 it also means […]<\/p>\n","protected":false},"author":570,"featured_media":3082,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[740],"tags":[473,729,728],"class_list":["post-3079","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-kotlin","tag-machine-learning","tag-onnx"],"acf":[],"blog_post_summary":"
Hello Android developers, Previously we looked at how to pre-process image inputs for an ONNX model using Kotlin. It\u2019s useful to understand this process because the principles apply to any model that you wish to use. On the other hand, having to write boilerplate code for input processing can be tedious \u2013 it also means […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/3079","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/users\/570"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/comments?post=3079"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/posts\/3079\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media\/3082"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/media?parent=3079"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/categories?post=3079"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-json\/wp\/v2\/tags?post=3079"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}