{"id":425,"date":"2022-09-29T09:53:57","date_gmt":"2022-09-29T16:53:57","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/windowsai\/?p=425"},"modified":"2022-09-29T09:53:57","modified_gmt":"2022-09-29T16:53:57","slug":"real-time-image-blurring-directx-resource-binding-in-the-windows-ml-samples-gallery","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/windowsai\/real-time-image-blurring-directx-resource-binding-in-the-windows-ml-samples-gallery\/","title":{"rendered":"Real-Time Image Blurring & DirectX Resource Binding in the Windows ML Samples Gallery"},"content":{"rendered":"

Last November, we announced the release of the <\/span>Windows ML Samples Gallery<\/span><\/a>. The newest release of the gallery adds 4 new samples, 2 of which are discussed in detail below.<\/span>\u00a0<\/span><\/p>\n

\u00a0<\/span><\/p>\n

Real-Time Background Image Blurring<\/span>\u00a0<\/span><\/h2>\n

Linnea May\u202f\u00a0<\/span><\/p>\n

\n

Real-time machine learning inference is a hot topic in ML, with applications such as real-time object detection in autonomous cars or background image blurring of a video stream during a work call. This new sample demonstrates the best practices to follow with Windows Machine Learning (Windows ML) and Microsoft Media Foundation to run background image blurring during inference on a real-time video stream at 30+ frames per second when using a dedicated GPU. <\/span>\u00a0<\/span><\/p>\n

\"Image<\/a><\/p>\n

Background image blurring running at 30 frames per second.\u00a0<\/span><\/i>\u00a0<\/span><\/p>\n

The sample uses an asynchronous <\/span>Media Foundation Transform<\/span><\/a> (MFT) to apply an effect to the input video stream.\u00a0<\/span>\u00a0<\/span><\/p>\n

The MFT will start by asynchronously requesting a new video sample from the capture source which in this case is the camera. This sample is converted into a shareable VideoFrame then run through the composition of three models: a preprocessing model, a Fully Convolutional Network, and a postprocessing model. Once the sample has been transformed, it\u2019s returned to the Media Foundation processing pipeline via an asynchronous callback to the preview sink, which displays the sample to the screen.\u00a0<\/span>\u00a0<\/span><\/p>\n

\"The<\/a>\u00a0<\/span><\/p>\n

The basic data flow of the Background Blur sample.<\/span><\/i>\u00a0<\/span><\/p>\n

Microsoft.AI.MachineLearningExperimental.LearningModelJoinOptions<\/span><\/a> fuses together three stages of the model:\u00a0<\/span>\u00a0<\/span><\/p>\n

    \n
  1. Preprocessing: The FCN needs 0-1 scaling and z-score normalization, so we use Windows ML\u2019s <\/span>LearningModelBuilder<\/span><\/a> to compose ONNX operators into a preprocessing stage.<\/span><\/li>\n
  2. FCN ResNet-50<\/span><\/a> takes in a preprocessed image of shape (N, 3, height, width), where N is the batch size, and outputs shape (N, 21, height, width) where each pixel is one-hot encoded as one of FCN-ResNet\u2019s 21 possible classes.<\/span><\/li>\n
  3. Postprocessing: We use another LearningModelBuilder to isolate and blur the pixels that are classified as \u201cbackground\u201d, then compose it with the non-blurred foreground.\u00a0\u00a0<\/span>\u00a0<\/span><\/li>\n<\/ol>\n

    Getting started<\/h4>\n

    Windows Machine Learning<\/span> Resources<\/span><\/p>\n\n\n\n\n\n\n\n\n
    Resource<\/span>\u00a0<\/span><\/td>\nDescription<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    Background Blur Sample Code<\/span><\/a>\u00a0<\/span><\/td>\nSource code for the background image blur sample<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    Windows ML Experimental API<\/span><\/a>\u00a0<\/span><\/td>\nDocumentation for the LearningModelBuilder and LearningModelJoinOptions<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    ORT model building unit tests<\/span><\/a>\u00a0<\/span><\/td>\nA good intro to building models with LearningModelBuilder and LearningModelJoinOptions<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    ONNX operators schema<\/span><\/a>\u00a0<\/span><\/td>\nONNX operators that can be used when building a model with LearningModelBuilder<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    ONNX Model Zoo FCN<\/span><\/a>\u00a0<\/span><\/td>\nOpen-source models in the ONNX format<\/span>\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

     <\/p>\n

    Media Foundation<\/span> Resources<\/span><\/p>\n\n\n\n\n\n\n
    Resource<\/span>\u00a0<\/span><\/td>\nDescription<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    Introduction to Media Foundation Transforms<\/span><\/a>\u00a0<\/span>\u00a0<\/span><\/td>\nStart here to learn more about Media Foundation Transforms as a model for processing media data<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    Windows Async MFT Sample<\/span><\/a><\/td>\nDemonstrates how to create an asynchronous Media Foundation Transform<\/span>\u00a0<\/span><\/td>\n<\/tr>\n
    Windows Capture Engine Sample<\/span><\/a><\/td>\nDemonstrates how to use the Media Foundation <\/span>CaptureEngine<\/span><\/a> to capture video<\/span>\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    \u00a0<\/span><\/p>\n

    DirectX Resource Binding<\/span>\u00a0<\/span><\/h2>\n

    Numfor Mbiziwo-Tiapo\u00a0<\/span><\/p>\n

    \n

    Learn how to bind and inference on Direct3D 12 Resources using ONNX Runtime in the <\/span>DX Resource Binding ORT Sample<\/span><\/a>.<\/span>\u00a0<\/span><\/p>\n

    \"Image<\/a><\/span><\/p>\n

    DX Resource Binding Sample<\/span><\/i>\u00a0<\/span><\/p>\n

    In this sample, images are drawn to the screen using Direct3D 12. The images are then preprocessed and inferenced on in real-time using the ONNX Runtime C++ API.<\/span>\u00a0<\/span><\/p>\n

    The general structure of the sample is stated below:<\/span>\u00a0<\/span><\/p>\n