{"id":429,"date":"2021-10-19T17:20:16","date_gmt":"2021-10-20T00:20:16","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/?p=429"},"modified":"2021-10-19T17:29:49","modified_gmt":"2021-10-20T00:29:49","slug":"build-ml-powered-experiences-with-windows-machine-learning-winml-and-intelligent-apis","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/build-ml-powered-experiences-with-windows-machine-learning-winml-and-intelligent-apis\/","title":{"rendered":"Build ML powered experiences with Windows Machine Learning (WinML) and Intelligent APIs!"},"content":{"rendered":"<p>Hello, fellow developers! Today I want to talk to you about <a href=\"https:\/\/docs.microsoft.com\/en-us\/windows\/ai\/windows-ml\/\">Windows Machine Learning<\/a> (WinML). I&#8217;ll also talk about how you can use Intelligent APIs, an abstraction over WinML to build Machine Learning powered experiences in your applications on Windows with very few lines of code.<\/p>\n<p>Machine Learning and Artificial Intelligence are definitely not unfamiliar terms anymore, with ML based features sneaking into almost every popular application we use everyday. Most of the ML workloads for these applications are deployed in cloud datacenters. In recent times though, with the steady improvement of processors and AI accelerators, machine learning on edge devices is becoming more and more commonplace.<\/p>\n<p>Windows Machine Learning (WinML) is a high-performance, reliable API for deploying hardware-accelerated ML inferences on Windows devices. It allows developers to use the capabilities of any Windows 10 (or higher) devices to perform model inference.<\/p>\n<h2>Hold on, what is model inference?<\/h2>\n<p>A machine learning model is an algorithm that learns from training data to identify patterns and perform predictions on real data. In order to create a machine learning model, it has to be \u201ctrained\u201d with training data. Once a model is created, it is then used on real data to calculate an output. This process of running real time data through a model to capture a prediction is called model inference.<\/p>\n<p>This is where WinML comes in. WinML is an API that enables developers to perform model inference by running the model on system hardware on Windows. It abstracts away a lot of the code you would need to otherwise write to interact with the hardware, perform ML computations and also optimize for performance.<\/p>\n<p>There are many benefits to using WinML, and performing ML inference on device in general. Here are some of them :<\/p>\n<ul>\n<li>Ease of development: WinML is built into all devices running Windows 10 and above, so all you really need to get started is a pre-trained model and Visual Studio to build that ML powered app everyone needs on Windows!<\/li>\n<li>Hardware support and Optimizations: With WinML, you only need to write code for your workload once and you can get hardware optimizations across CPUs, GPUs and AI accelerators with consistent behavior.<\/li>\n<li>Low Latency, Real-Time Results: Since all the ML computations are performed on device versus the cloud, there\u2019s little to no latency even for real-time analysis of large quantities of data. This can be extremely valuable in applications that require real time results like performance-intensive games.<\/li>\n<li>Offline Experience and Flexibility: Running your machine learning models on device requires no internet. This means your ML features can even work offline!<\/li>\n<li>Security: All your data remains on your device. This keeps the data extremely secure, incredibly useful for applications that involve sensitive data management.<\/li>\n<\/ul>\n<h2>ONNX models<\/h2>\n<p>We mentioned earlier that in order to use WinML, you need to have an existing model. This model needs to be in <a href=\"https:\/\/onnx.ai\/\">ONNX<\/a> format. ONNX is an open format built to represent machine learning models that can work cross-platform and is tool\/framework agnostic.<\/p>\n<p>How does one obtain an ONNX model?<\/p>\n<p>Most of the common ML use-cases have state-of-the-art models available for you to choose from! The <a href=\"https:\/\/github.com\/onnx\/models\">ONNX model zoo<\/a> is typically the one-stop shop for any such model.<\/p>\n<p>You can also leverage existing ML frameworks like <a href=\"https:\/\/www.tensorflow.org\/\">Tensorflow<\/a> or <a href=\"https:\/\/pytorch.org\/\">Pytorch<\/a> to train your model, and then export it to ONNX.<\/p>\n<p>You could also use tools like <a href=\"https:\/\/customvision.ai\">Custom Vision<\/a> that offer no-code experiences to create image classification or object detection models.<\/p>\n<h2>Intelligent APIs<\/h2>\n<p>Now that we\u2019ve talked about WinML and ONNX models, let me tell you that we are working on a project that makes building ML powered applications on Windows even easier! This project is called <a href=\"https:\/\/github.com\/CommunityToolkit\/Labs-IntelligentAPIs\">Intelligent APIs<\/a>, the goal of which is to make machine learning tasks easier for developers to leverage in their applications without needing ML expertise or creating a new model. By just importing a nuget package and calling a function, we want developers to be able to build intelligent app experiences without needing to deal with the complexities of using machine learning models on Windows.<\/p>\n<p>So if you want to classify an image in your application, all you have to do is call a function in a single line of code. Easy peasy!<\/p>\n<p>So far we have released two nuget packages that you can easily install into your application and start using.<\/p>\n<ul>\n<li><code>CommunityToolkit.Labs.Intelligent.ImageClassification<\/code>: This package exposes an API that performs Image Classification using <a href=\"https:\/\/github.com\/onnx\/models\/tree\/master\/vision\/classification\/squeezenet\">SqueezeNet<\/a> which can detect <a href=\"https:\/\/github.com\/onnx\/models\/blob\/master\/vision\/classification\/synset.txt\">1000<\/a> different classes.<\/li>\n<li><code>CommunityToolkit.Labs.Intelligent.ObjectDetection<\/code>: This package exposes an API that performs Object Detection and Segmentation using <a href=\"https:\/\/github.com\/hunglc007\/tensorflow-yolov4-tflite\">YOLOv4<\/a>. This model can detect objects of <a href=\"https:\/\/github.com\/hunglc007\/tensorflow-yolov4-tflite\/blob\/9f16748aa3f45ff240608da4bd9b1216a29127f5\/android\/app\/src\/main\/assets\/coco.txt\">80<\/a> different classes and also returns the co-ordinates of their location in the image.<\/li>\n<\/ul>\n<h3>Getting started with Intelligent APIs<\/h3>\n<p>Let&#8217;s go through installing and using the package that performs Image Classification.<\/p>\n<p>To perform image classification, import the <code>CommunityToolkit.Labs.Intelligent.ImageClassification<\/code> nuget package.To use the <code>ClassifyImage<\/code> API, you will need to pass a StorageFile object which is the image file itself, and the number of top results that you want (optional) as arguments.<\/p>\n<p>We&#8217;ll now walk through a step-by-step guide of how you can start using the nuget package that performs image classification in your UWP application.<\/p>\n<ol>\n<li>Open your UWP project on Visual studio.<\/li>\n<li>Right click on your project in the Solution Explorer and click on \u201cManage Nuget Packages\u201d.\n<p align=\"center\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-content\/uploads\/sites\/61\/2021\/10\/manage.jpg\" alt=\"Manage Nuget Packages\" width=\"300\" \/><\/p>\n<\/li>\n<li>Add a package source with the URL as https:\/\/pkgs.dev.azure.com\/dotnet\/CommunityToolkit\/_packaging\/CommunityToolkit-Labs\/nuget\/v3\/index.json<\/li>\n<\/ol>\n<p align=\"center\"><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-content\/uploads\/sites\/61\/2021\/10\/packagesources.jpg\" alt=\"Add a Package Source\" width=\"300\" \/><\/p>\n<ol start=\"4\">\n<li>In this new source, there will be two nuget packages listed: <code>CommunityToolkit.Labs.Intelligent.ImageClassification<\/code> and <code>CommunityToolkit.Labs.Intelligent.ObjectDetection<\/code>. For this example let&#8217;s install <code>CommunityToolkit.Labs.Intelligent.ImageClassification<\/code> in our project.<\/li>\n<\/ol>\n<p>Now that the package has been installed, let\u2019s use it in our app!<\/p>\n<p>Let\u2019s create a very simple button in our XAML file that opens up a filepicker so you can select a file that you want to be classified.<\/p>\n<pre><code class=\"language-html\">&lt;button\r\n  name=\"ButtonRun\"\r\n  ToolTipService.ToolTip=\"Pick and evaluate and image\"\r\n  Width=\"100\"\r\n  Height=\"40\"\r\n  Click=\"ButtonRun_Click\"\r\n  IsEnabled=\"True\"\r\n  VerticalAlignment=\"Bottom\"\r\n  RenderTransformOrigin=\"2.085,7.842\"\r\n  Margin=\"84,0,0,654\"\r\n&gt;\r\n  &lt;Button.Content&gt; Pick image &lt;\/Button.Content&gt;\r\n&lt;\/button&gt;<\/code><\/pre>\n<p>Go to the code-behind C# file for the XAML to define the <code>ButtonRun_Click<\/code> function that will be called on-click of the button we just created.<\/p>\n<p>We need to import the namespace <code>CommunityToolkit.Labs.Intelligent.ImageClassification<\/code>, so at the top of the file, along with the other imports, add this line:<\/p>\n<pre><code class=\"language-cs\">    using CommunityToolkit.Labs.Intelligent.ImageClassification;<\/code><\/pre>\n<p>Next, create the ButtonRun_Click function that is called on click of the button.<\/p>\n<pre><code class=\"language-cs\">    private async void ButtonRun_Click(object sender, RoutedEventArgs e){}<\/code><\/pre>\n<p>Inside this function, let\u2019s create a FilePicker.<\/p>\n<pre><code class=\"language-cs\">private async void ButtonRun_Click(object sender, RoutedEventArgs e)\r\n{\r\n    FileOpenPicker fileOpenPicker = new FileOpenPicker();\r\n    fileOpenPicker.SuggestedStartLocation = PickerLocationId.PicturesLibrary;\r\n    fileOpenPicker.FileTypeFilter.Add(\".jpg\");\r\n    fileOpenPicker.FileTypeFilter.Add(\".png\");\r\n    fileOpenPicker.ViewMode = PickerViewMode.Thumbnail;\r\n    StorageFile selectedStorageFile = await fileOpenPicker.PickSingleFileAsync();\r\n}\r\n<\/code><\/pre>\n<p>The above lines of code initialize a filepicker, restrict the files to be images and get the selected file stored in a StorageFile object called selectedStorageFile.<\/p>\n<p>Now, all we have to do is call the ClassifyImage function and pass this file in to get the image classification result!<\/p>\n<pre><code class=\"language-cs\">    var list = await SqueezeNetImageClassifier.ClassifyImage(selectedStorageFile, 3);<\/code><\/pre>\n<p>The list stores the top 3 classification results based on confidence that is determined by the SqueezeNet model.\nThat\u2019s all you need to do to get the ML part working. Now you can display this result or use it for anything your creativity comes up with!<\/p>\n<p>Wasn\u2019t that easy?<\/p>\n<p>You can access the complete sample on <a href=\"https:\/\/github.com\/beth-panx\/win-student-dev\/tree\/main\/samples\/WinML-IntelligentAPI\">GitHub<\/a><\/p>\n<p>The two packages that are available right now are just the start of what Intelligent API aims to offer. The idea is to make machine learning model evaluation easier on windows and enabling developers to seamlessly create intelligent experiences on Windows without needing any domain specific knowledge. This is an open source contribution, so feel free to create issues for new ideas, or build your own packages and create a pull request!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article introduces Windows Machine Learning, an API for machine learning model inference on Windows, and then talks about a new initiative called Intelligent APIs that aims to make it easier to integrate ML workloads into modern day applications.<\/p>\n","protected":false},"author":54489,"featured_media":208,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[17,41,42,5,8],"class_list":["post-429","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ifdef-windows","tag-ai","tag-ml","tag-onnx","tag-uwp","tag-windows"],"acf":[],"blog_post_summary":"<p>This article introduces Windows Machine Learning, an API for machine learning model inference on Windows, and then talks about a new initiative called Intelligent APIs that aims to make it easier to integrate ML workloads into modern day applications.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/posts\/429","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/users\/54489"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/comments?post=429"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/posts\/429\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/media\/208"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/media?parent=429"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/categories?post=429"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ifdef-windows\/wp-json\/wp\/v2\/tags?post=429"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}