Model inputs<\/h2>\n
\n Pre-trained models in formats that can be shared across platforms are incredibly powerful, but it makes sense that each model must have inputs supplied in a known and repeatable way to get the most accurate results. Models will typically specify the expected format of their input parameters.\n<\/p>\n
\n FOr example, on the information page for the MOBILENET V2 model<\/a>, (from last week’s sample) you\u2019ll find the following information along with sample Python code that shows an example of how to pre-process image data before sending to the model.\n<\/p>\n All pre-trained models expect input images normalized in the same way, i.e. mini-batches of 3-channel RGB images of shape\u00a0(3 x H x W), where\u00a0H\u00a0and\u00a0W\u00a0are expected to be at least\u00a0224. The images have to be loaded in to a range of\u00a0[0, 1]\u00a0and then normalized using\u00a0mean = [0.485, 0.456, 0.406]\u00a0and\u00a0std = [0.229, 0.224, 0.225].<\/p><\/blockquote>\n \n You can use Python notebook to interactively step through<\/a> the pre-processing code and test the model, such as with this sample image input:\n<\/p>\n \n Many clients (including mobile apps on Android) can\u2019t easily re-use the Python code provided, and will have to implement their own pre-processing in a native language like Java or Kotlin. The following sections show examples of how the Python code can be adapted to interact with an ONNX model on Android using Kotlin.\n<\/p>\n \n The sample does its image pre-processing in the \n The first step is resizing to the required dimensions (224×224) with this function call:\n<\/p>\n \n Further manipulation is done in the \n You\u2019ll find some additional image processing code for bitmap conversion to the correct format for ONNX in the ImageUtil.kt<\/a><\/strong> file.\n<\/p>\n \n The model page<\/a> sample code also contains information about how to parse the output:\n<\/p>\n # Tensor of shape 1000, with confidence scores over Imagenet’s 1000 classes\n \n This means the result is a thousand \u2018confidence scores\u2019, one for each possible classification. The model result doesn\u2019t include the text descriptions of each possible classification, so as a consumer of the model you need to separately download the classes list<\/a> and match them up with the scores. \n<\/p>\n \n You can find the code that extracts the scores and runs them through the custom functions \n To display the results with the correct label, in MainActivity.kt<\/a><\/strong> the downloaded file (placed in the raw resources directory) is parsed:\n<\/p>\n \n And when the results are displayed, the index of the highest scoring classifications are used to show the correct text value from the list:\n<\/p>\n \n Any model you decide to add to your apps will have its own input parameter formatting requirements and output parsing rules, so remember to check the model\u2019s documentation while implementing. \n<\/p>\n \n Not all models have images as their input or collections of data as the output. For a simpler example, this linear regression sample on GitHub<\/a> shows how to provide a simple numerical value to a model which returns a single numerical result (check the associated blog<\/a> for instructions to build, convert, and place the model in the Android code).\n<\/p>\n \n More information about the ONNX Runtime is available at\u00a0onnxruntime.ai<\/a>\u00a0and also on\u00a0YouTube<\/a>.\n<\/p>\n \n If you have any questions about applying machine learning, or would like to tell us about your apps, use the\u00a0feedback forum<\/a>\u00a0or message us on\u00a0Twitter @surfaceduodev<\/a>.\n<\/p>\n![\"\"](\"https:\/\/devblogs.microsoft.com\/surface-duo\/wp-content\/uploads\/sites\/53\/2023\/02\/onnx-python.png\")
<\/a><\/p>\nImage input formatting on Android<\/h2>\n
analyze<\/code> function in ORTAnalyzer.kt<\/a>.\n<\/p>\nBitmap.createScaledBitmap(it, 224, 224, false)<\/pre>\n
preProcess<\/code> function which is defined in ImageUtil.kt<\/a>. In this code snippet you can see the normalization process on each pixel that matches the parameters specified for the model:\n<\/p>\nbitmap.getPixels(bmpData, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height)\r\nfor (i in 0..IMAGE_SIZE_X - 1) {\r\n for (j in 0..IMAGE_SIZE_Y - 1) {\r\n val idx = IMAGE_SIZE_Y * i + j\r\n val pixelValue = bmpData[idx]\r\n imgData.put(idx, (((pixelValue shr 16 and 0xFF) \/ 255f - 0.485f) \/ 0.229f))\r\n imgData.put(idx + stride, (((pixelValue shr 8 and 0xFF) \/ 255f - 0.456f) \/ 0.224f))\r\n imgData.put(idx + stride * 2, (((pixelValue and 0xFF) \/ 255f - 0.406f) \/ 0.225f))\r\n }\r\n}<\/pre>\nOutput parsing<\/h2>\n
softMax<\/code> and getTop3<\/code> to get the indices of the highest scoring classifications in ORTAnalyzer.kt<\/a><\/strong>:\n<\/p>\nval rawOutput = ((output?.get(0)?.value) as Array<FloatArray>)[0]\r\nval probabilities = softMax(rawOutput)\r\nresult.detectedIndices = getTop3(probabilities)\r\n<\/pre>\n
resources.openRawResource(R.raw.imagenet_classes).bufferedReader().readLines()<\/pre>\n
detected_item_1.text = labelData[result.detectedIndices[0]]\r\ndetected_item_value_1.text = \"%.2f%%\".format(result.detectedScore[0] * 100)<\/pre>\n
Resources and feedback<\/h2>\n