{"id":2175,"date":"2015-07-21T16:34:28","date_gmt":"2015-07-21T23:34:28","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/index.php\/2015\/07\/21\/using-camera-stream-for-real-time-object-tracking-in-windows-apps\/"},"modified":"2020-03-18T23:20:51","modified_gmt":"2020-03-19T06:20:51","slug":"using-camera-stream-for-real-time-object-tracking-in-windows-apps","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/using-camera-stream-for-real-time-object-tracking-in-windows-apps\/","title":{"rendered":"Using camera stream for real time object tracking in Windows apps"},"content":{"rendered":"

Summary<\/h2>\n

The imaging and computing capabilities in commodity mobile devices provide exciting possibilities for computer vision based solutions. While some computer vision scenarios are purpose-built from the get-go with mobile device usage in mind, we have also witnessed several scenarios where existing legacy solutions could benefit from replacing the often expensive and immobile hardware (camera and computing) setup with modern smartphones and their imaging capabilities.<\/p>\n

As an example of this, we ran a pilot with a leading sports brand to enable tracking and analyzing object movement \u2013 consider a ball leaving a bat or a club at arbitrary launch angles, high velocity and varying spin rates.<\/p>\n

The partner wanted to explore how well modern smartphones would suit their needs and enable moving away from the usage patterns and restrictions caused by their existing imaging hardware and analytics pipeline.<\/p>\n

As a byproduct of the pilot, we identified reusable patterns and ways to avoid common pitfalls when implementing object tracking in Windows apps. This case study, the referenced blog and sample code will introduce our key findings and help the reader identify the best approaches to object detection and tracking to use in their Windows apps.<\/p>\n

Solution<\/h2>\n

The scenario we faced could be broken down to this pipeline.<\/p>\n

    \n
  1. Capture video data from a Windows phone<\/li>\n
  2. Understand the data format provided to us<\/li>\n
  3. Identify the object we\u2019re interested in tracking from the video data<\/li>\n
  4. Lock and track the object while it blazes through the field of view<\/li>\n
  5. Analyze what actually happened to the object<\/li>\n
  6. Don\u2019t mess up the performance during steps 1-5. Easy.<\/li>\n<\/ol>\n

    Capture video data from a Windows device<\/h2>\n

    Capturing video frames in Windows apps has been well documented in Windows platform documentation. Use these MSDN resources to understand how to get the video data:<\/p>\n

    Quickstart: Capturing video by using the MediaCapture API<\/a><\/p>\n

    Media capture using capture device sample app<\/a><\/p>\n

    Understand the data format<\/h2>\n

    Before processing the data, we must first understand the data characteristics (format and frequency). Here is a sample of the image data that we are processing.<\/p>\n

    \"Target<\/p>\n

    A four pixel section in NV12 image data – location of the bytes in NV12 byte array A, where h is the height of the image and w is the width of the image (in bytes).<\/em><\/p>\n

    In the first part of Tomi Paananen\u2019s blog posts, he introduces two common YUV color space formats<\/a> and then dives into details on how to work with NV12 format, which is a conventional data type for smartphone cameras.<\/p>\n

    Identify the object we\u2019re interested in tracking from the video data<\/h2>\n

    Now that we understand the type of data provided to us, it\u2019s time to think about how to identify the objects we\u2019re looking for from the video data.<\/p>\n

    \"Object<\/p>\n

    Red areas filtered out from a video frame using the problem solving pipeline described below.<\/p>\n

    The problem solving pipeline described below is broken down in detail in a blog post that explains approaches to detecting areas of similar pixels using threshold values and applying versions of the convex hull algorithm to ease detection of objects<\/a>.<\/p>\n\n\n\n\n\n\n\n\n\n
    What is the result?<\/th>\nHow do we get there?<\/th>\n<\/tr>\n<\/thead>\n
    An object, with a specific shape, identified in the image (location, size, shape etc.)<\/td>\nFind the center of mass of the two dimensional object and its boundary.<\/td>\n<\/tr>\n
    Boundary of the detected object.<\/td>\nApply convex hull algorithm to object map (extended binary image, where the background is removed).<\/td>\n<\/tr>\n
    Object map, where all significant (suspected)<\/td>\nobjects are separated and the background is removed. Individualize objects from a binary image.<\/td>\n<\/tr>\n
    Binary image in which objects have value 1 and background value 0.<\/td>\nApply algorithm to extract objects with some criteria from the background. E.g. chroma filter.<\/td>\n<\/tr>\n
    Chroma filter implementation to extract objects from image.<\/td>\nStart coding!<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Lock and track the object while it moves through the field of view<\/h2>\n

    Since our object tracking solution is required to provide real-time analysis while running on (Windows) mobile devices, we need to be considerate on what algorithms are used and on which set of pixels from a frame to actually apply them.<\/p>\n

    Instead of applying all methods described in the previous section on each analyzed frame, our sample implementation provides a way to lock the object of interest and, in following frames, apply chroma filtering to only the small subset of frame pixels which contain the object we want to track.<\/p>\n

    \"VaatturiLockedToObjectScaled\"<\/p>\n

    Target object locked, and tracking limited to the region marked by the green rectangle<\/p>\n

    This blog post explains the object locking and partial pixel map tracking<\/a> in more detail.<\/p>\n

    Analyze what actually happened to the object<\/h2>\n

    Unsurprisingly, our scenario eventually ends up in a scenario very common in technology and life: We have raw and interesting data \u2013 what should we actually do to analyze it?<\/p>\n

    \"VaatturiObjectMotionCapturedScaled\"<\/p>\n

    Object (USB cannon projectile wrapped with pink sticker) motion captured<\/p>\n

    Tomi Paananen and Juhana Koski provide some concrete tips for the (nearly) real time analysis \u2013 explaining<\/p>\n

    a) How a ring buffer approach to buffer video was used in our solution to enable a good compromise on consumed resources and provide ability to do a bit of post processing on the captured video frames<\/p>\n

    b) Iterating through the buffered video frames to understand object displacement<\/a> using the pipeline mentioned earlier in this post<\/p>\n

    In our scenario, further data analysis could contain for example real time speed, launch angle and rotation related calculations \u2013 and perhaps pushing that information to a secondary, non-real time, analytics pipeline that could utilize Azure Machine Learning or other approaches to get more insight or predictions.<\/p>\n

    Next Steps<\/h2>\n

    We still have ways to go in reaching our stretch goal of analyzing moving objects at a high frame rate (> 400 fps) using mobile devices. The next areas that we would focus on to continue to improve performance of the pipeline under challenging conditions such as variations in lighting, distortion of objects shape and color blending after a heavy impact causing high velocity.<\/p>\n

    Reusable assets<\/h2>\n

    Tracking objects from video feed \u2013 3 part blog series<\/p>\n