![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2015\/07\/image001-1.png\")
<\/p>\n
Figure 1. Conceptual Goal of Project<\/p>\n
Overview of the Solution<\/h2>\n
Project \u201cDolittle\u201d initially leveraged the open source computer vision library OpenCV (http:\/\/www.opencv.org<\/a>). OpenCV supports image detection, object recognition, various machine learning algorithms, classifiers, and video analysis, among other capabilities. Initial work focused on training OpenCV\u2019s Haar cascades to recognize one dog, in this case, a Smooth-Haired Collie named \u201cMici\u201d.<\/p>\n Figure 2. Mici<\/p>\n OpenCV uses a relatively typical training methodology. It receives a large number of annotated pictures as input. Here the developer took roughly 400 pictures of Mici and hand-annotated them; the training process required approximately a week of processing time. This approach was deemed inappropriate, partly because of the training time and because such training would be required on a per dog basis! While it is likely that the training time could have been accelerated by the use of Azure scale, we believed this approach would ultimately not be flexible enough to support the infinite variation in dog shape and dog movement.<\/p>\n Nevertheless, object detection \u2013 especially moving objects such as animals \u2013 poses a formidable problem. The Kinect receives (through its IR camera) a depth stream in addition to the 30fps HD RGB color data.1<\/a> However, these constitute simply a point cloud that requires sophisticated analysis in order recognize a shape in real time. The current Kinect retail product presently leverages a massive amount of machine learning to perform human skeletal tracking.<\/p>\n Various other approaches were considered including Berkeley\u2019s image processing machine learning library Caffe (http:\/\/caffe.berkeleyvision.org\/<\/a>), internal Kinect code, and others.<\/p>\n Eventually, the team collaborated with a group in Microsoft Research, which had built software for very high-resolution, real-time hand tracking. Unlike other approaches, MSR used both<\/em> machine learning and<\/em> model fitting in real time to identify and track hands with extremely high fidelity. Supplementing machine learning with the ability to match a 3D mesh against the Kinect video stream enabled considerably more accuracy, as shown in the video located here<\/a>. See as well the hand tracking paper presented at SIGCHI 2015 here<\/a>.<\/p>\n Figure 3. Hand Tracking Video<\/p>\n However, the problem of dog tracking (and by extension similar large objects that move) has some differences with hand tracking:<\/p>\n A multi-stage pipeline is used to recognize an object. The process matches predefined depth-aware \u201cposes\u201d (approximately 100,000 of them)2<\/a> to what the Kinect sees. To do this matching, a jungle ML algorithm3<\/a> is used to detect a set of candidate poses and then a particle swarm optimization algorithm performs \u201cmodel fitting\u201d. In model fitting, the observed image is matched against prebuilt poses and the best choice is selected.<\/p>\n <\/p>\n Figure 4. Model Fitting Visualization<\/p>\n To create the poses, a \u201crigged\u201d (articulated) Blender 3D model, such as the hand model below, is used:<\/p>\n Figure 5. Rigged Hand Model in Blender<\/p>\n (\u201cRigged\u201d meaning that the 3D mesh includes \u201cbones\u201d and the model can be articulated.) Poses of the model rotated in 3d space can be used to test the recognition. These models are then rotated and articulated to get the thousands of poses that will be used to \u201cmodel fit\u201d against the observed Kinect data.<\/p>\n An example of a dog model (a border collie) in Blender is shown below:<\/p>\n Figure 6. Border Collie in Blender<\/p>\n These models can be used as the basis for building poses which the ML algorithms will use to track the dog in question.<\/p>\n Initially, the goal of this project was modest: to track one or two specific dogs (Ilkka\u2019s dog Mici and Barry\u2019s dog Joe). Later the project will be able to recognize and tune for different sizes and breeds, using machine learning and likely leveraging training data sets such as the Stanford Dogs Dataset (an annotated library of some 22,000 dog images \u2013 http:\/\/vision.stanford.edu\/aditya86\/ImageNetDogs\/<\/a> ) and potentially using such techniques as model deformation to match different breeds.<\/p>\n We have been able to track Ilkka\u2019s dog Mici in real time, with some limitations (see the TED Case Study entitled \u201cBackground and Floor Removal from Depth Camera Data\u201d for a discussion of one of the thorniest issues). The next step was to improve the tracking and then move on to identify features on the dog \u2013 ear position, tail wag rate, and so forth \u2013 in order to infer the dog\u2019s state of mind. In addition, work is also under way to use machine learning algorithms to decipher dog vocalizations; these topics and others will be discussed in future case studies.<\/p>\n OpenCV is at http:\/\/www.opencv.org<\/a> .<\/p>\n The Kinect SDK is at http:\/\/www.microsoft.com\/en-us\/kinectforwindows\/<\/a><\/p>\n We believe this project has numerous possibilities for reuse. The most significant accomplishment of the project to date is to demonstrate that real time, very high fidelity reconstruction (far more significant than 3D cameras out of the box) is possible. Further, since the model fitting uses 3D models that can be tagged, feature extraction (e.g., ear position) is made possible, and this capability leads to a number of important scenarios.<\/p>\n It should be possible, using the code base, to extend the recognition to other animals such as cats (actually, a much easier problem given that physical variation among cat breeds is substantially less than for dogs) and possibly horses.<\/p>\n The resolution of the tracking enables scenarios previously difficult or impossible for Kinect. For example, 24-hour scanning of premature human babies is, with work, feasible (as the out of the box Kinect has certain minimum size limitations); in addition, similar scanning for normal babies (for terrified new parents) could also be done and there are a number of similar scenarios. Finally, such technology could be used for the benefit of Alzheimer\u2019s and other physically challenged individuals.<\/p>\n <\/a> For more specifics on the Kinect, see here<\/a>; for the Kinect SDK, see here<\/a>.<\/p>\n <\/a> Initial set. One of the goals of the project is to see if we can match with substantially fewer prebuilt poses.<\/p>\n <\/a> For more on decision jungles, see here<\/a>. To quote: \u201cUnlike conventional decision trees that only allow one path to every node, a DAG [Directed Acyclic Graph] in a decision jungle allows multiple paths from the root to each leaf.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":" With the arrival of commodity depth-capable cameras, specifically, the Microsoft Kinect, as well as high-performance machine learning algorithms, entirely new capabilities are made possible. Working with academic experts and others, we are attempting to track the movements, body language, and vocalizations of dogs.<\/p>\n","protected":false},"author":21345,"featured_media":11215,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[19],"tags":[127,226,239,253,282],"class_list":["post-2184","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-computer-vision","tag-kinect-sdk","tag-machine-learning-ml","tag-microsoft-kinect","tag-opencv"],"acf":[],"blog_post_summary":" With the arrival of commodity depth-capable cameras, specifically, the Microsoft Kinect, as well as high-performance machine learning algorithms, entirely new capabilities are made possible. Working with academic experts and others, we are attempting to track the movements, body language, and vocalizations of dogs.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21345"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=2184"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2184\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/11215"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=2184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=2184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=2184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2015\/07\/image002-1.jpg\")
<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2015\/07\/image003-1.jpg\")
<\/p>\n\n
![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2015\/07\/image005-2.jpg\")
<\/p>\n![\"Image](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2015\/07\/image006-1.jpg\")
<\/p>\nCode Artifacts<\/h2>\n
Opportunities for Reuse<\/h2>\n
\n