{"id":28493,"date":"2016-11-04T11:26:16","date_gmt":"2016-11-04T18:26:16","guid":{"rendered":"https:\/\/blog.xamarin.com\/?p=28493"},"modified":"2016-11-04T11:26:16","modified_gmt":"2016-11-04T18:26:16","slug":"microsoft-research-ships-intelligent-apps-with-the-power-of-c-and-ai","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/xamarin\/microsoft-research-ships-intelligent-apps-with-the-power-of-c-and-ai\/","title":{"rendered":"Microsoft Research Ships Intelligent Apps with the Power of C# and AI"},"content":{"rendered":"<div class=\"alignright\"><video width=\"250\" controls loop autoplay>\nYour browser does not support the video tag.<\/video><\/div>\n<p><a href=\"https:\/\/www.microsoft.com\/en-us\/research\/product\/microsoftpix\/\" target=\"_blank\">Microsoft Pix<\/a> turns iPhone users into better photographers, using artificial intelligence and machine learning to automatically capture the best shots. Even before a user presses the shutter button, Microsoft&#8217;s rich computational photography algorithms make key adjustments, including: optimizing for lighting, recognizing and focusing on faces, and automatically selecting the best shots from a &#8220;burst&#8221; of photos.<\/p>\n<p>Microsoft Pix is the result of decades of computational photography research across several product and research teams. Today, we&#8217;ve invited Yung-Shin Lin, Principal Software Engineering Manager, and Josh Weisberg, Principal PM, to share how they got started with mobile development, and how Xamarin for Visual Studio and HockeyApp help them innovate at a rapid pace.<\/p>\n<p><strong>Tell us a little bit about Microsoft Research, your team, and your role. Have you always been a developer?<\/strong><\/p>\n<p><em>Josh Weisberg<\/em>: Our team, the Computational Photography team, is part of Microsoft Research\u2019s (MSR) Cognitive Services group. Our team is unique: early last year, we transitioned from researchers who build apps to showcase their research to a cross-functional Research &amp; Development team. Our PMs, developers, and designers work closely with our researchers to create apps that deliver unique and compelling functionality with simple user interfaces. I\u2019m actually not a developer, but I work closely with our developers, engineers, designers, and UX researchers to bring these apps to life.<\/p>\n<p><em>Yung-Shin Lin<\/em>: We are a small, but agile, team. In addition to Josh, our team is made up of one designer, five researchers, and seven developers. Personally, I\u2019ve been a developer in the Microsoft Windows organization for 18 years &#8211; as a software engineer, development lead, and I joined Microsoft Research last year with my current role, engineering manager. I\u2019ve enjoyed the power of .NET and C#, beginning when I worked on the first version of .NET framework APIs.<\/p>\n<p><strong>Tell us about the app and what prompted your team to build it. What were your goals?<\/strong><\/p>\n<p><em>JW<\/em>: Today\u2019s phones take pretty good pictures, but there are many situations where they don\u2019t. We realized that there was an opportunity to utilize Artificial Intelligence (AI) and Machine Learning (ML), combined with continuous frame buffering, to teach a camera how to make better photos.<\/p>\n<p>In v1, we focused on people, which we believe are the most important subjects. When Pix detects a person in a photo, AI kicks in, telling it that the person is the most important element in the photo. This begins with the capture settings, and then extends to picking out the best frame.<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/SideBySide-1.jpg\" alt=\"sidebyside-1\" width=\"600\" class=\"aligncenter size-full wp-image-28507\" \/>\n<strong>How does it fit into your&nbsp;or your department\u2019s&nbsp;mobile strategy?<\/strong><\/p>\n<p><em>JW<\/em>: For many years, Microsoft\u2019s researchers have developed very sophisticated technology that enables new photo and video scenarios. We want to create apps that enable users to take advantage of this advanced technology and techniques, without needing to be computer science or photography experts.<\/p>\n<p>In other words, our goal is to empower our customers, bringing Microsoft \u2018s computational photography to as many users as possible. Since most consumers use their mobile phones for photography, our primary focus is these devices.<\/p>\n<p><strong>Walk us through the app \u2013 what does it allow users to do?<\/strong><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/Family-1.gif\" alt=\"family-1\" width=\"200\" class=\"alignright size-full wp-image-28508\" \/><em>JW<\/em>: Microsoft Pix helps you capture the best picture every time you push the button.\u202fWhen Machine Learning is at its best, we see the results, not the effort.\u202fYou might recognize some of the capabilities we\u2019ve brought from <a href=\"http:\/\/research.microsoft.com\/en-us\/um\/redmond\/projects\/hyperlapseapps\/\" target=\"_blank\">Microsoft\u202fHyperlapse<\/a>,\u202fMicrosoft Selfie, Blink (now Live Image), and other computational photography projects.<\/p>\n<p>When a user takes photo, Microsoft Pix takes a burst of photos to make sure nothing is missed, selects best images from the burst, and tweaks lighting, contrast, and focus to suggest the best option. When Microsoft Pix detects faces, it automatically adjusts exposure to help faces look better, even in difficult lighting situations. It also uses proprietary multi-frame denoise technology, which uses other burst frames to enhance the Best Image, so faces and backgrounds are as true-to-life as possible. Microsoft Pix also optimizes videos: if the app detects any motion, it creates a &#8220;Live Image&#8221; (looping video), and helps users stabilize and time lapse existing videos.<\/p>\n<p><strong>Why did your team choose Xamarin?<\/strong><\/p>\n<p><em>JW<\/em>: We wanted to deliver Microsoft Pix across multiple mobile platforms, and based on our experience <a href=\"https:\/\/blogs.msdn.microsoft.com\/vcblog\/2015\/06\/26\/hyperlapse-cross-platform-mobile-development-with-visual-c-and-xamarin\/\" target=\"_blank\">using Xamarin for Hyperlapse<\/a>, it was an easy decision to choose Xamarin. Many of our developers have worked on other Microsoft products, so they have considerable C# experience, and Xamarin helped them get started faster.<\/p>\n<p><em>YL<\/em>: We knew that we&#8217;d get a huge productivity gain from both from using our C# knowledge and sharing architecture and code across different mobile platforms.<\/p>\n<p>We\u2019re very happy about the flexibility and reusability we\u2019ve seen with Xamarin. We create a common business logic layer that\u2019s shared across platforms, and we\u2019ve designed a platform-dependent layer to tailor the UI to match users\u2019 expectations and to leverage platform-specific APIs \u2013 like photo capture and gallery. Due to the ease of use for .NET P\/Invoke, we can easily \u201ctalk\u201d to our algorithm, written in high performant C++ code, and we use this same C++ code across all platforms.<\/p>\n<p><strong>What do you think about when you hear \u201cnative app\u201d?<\/strong><\/p>\n<p><em>YL<\/em>: For me, \u201cnative\u201d isn\u2019t about the language or framework used to develop the app. It\u2019s more important that the app fits the distinct personality of the operating system and performs like any other app on the platform.<\/p>\n<p>For Microsoft Pix, we want a user interface that\u2019s familiar to the users, and we want to achieve high performance, even with AI\u2019s heavy computation needs.<\/p>\n<p><strong>How long did it take to ship the app, from design to deploy?<\/strong><\/p>\n<p><em>JW<\/em>: As the PM, I developed the idea over time, starting in 2014. I completed the specs and preliminary designs in late summer 2015, and we started development in December 2015. From a coding perspective, it took about 7 months.<\/p>\n<p><em>YL<\/em>: I\u2019m pretty amazed at how fast the team shipped the first release \u2013 it took the team just 7 months. We wrote our first line of Microsoft Pix mid-January, and, due to a combination of our team&#8217;s agility, hard work, existing C# experience, and the power of Xamarin, we got to market extremely quickly, launching the app in July of this year.<\/p>\n<p><strong>How do you use HockeyApp?<\/strong><\/p>\n<p><em>YL<\/em>: HockeyApp is indispensable in our daily process. We want to continuously improve our quality and reliability for our users, and, during internal releases for dogfood users, we closely monitor HockeyApp crash reports and use it to automatically file bugs in Visual Studio. We prioritize and constantly fix the top 10 crashes \u2013 and continue to do so after we ship and gather real-world crash data. After our first few updates, we\u2019ve drastically improved our reliability and quality.<\/p>\n<p><strong>What\u2019s your team planning to build next?<\/strong><\/p>\n<p><img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/44\/2019\/03\/Weekend-1.gif\" alt=\"weekend-1\" width=\"300\" class=\"alignright size-full wp-image-28509\" \/><em>JW<\/em>: We\u2019ve successfully shipped the iOS version of Microsoft Pix and are now actively working on the Android version. We\u2019ve released 5 updates to the iOS version (as of September), and will continue to add new features and fix bugs.<\/p>\n<p><em>YL<\/em>: I\u2019m very excited to work on the Android version of Microsoft Pix, using our cross-platform design and shared code. Xamarin\u202fhas\u202fhelped us get our iOS version to market quickly, and, with our Android version, we\u2019ll be able to reuse much of the codebase and take advantage of cross-platform design. Since we don\u2019t have to start from scratch or work in another different language, we hope to deliver our Android version soon.<\/p>\n<p><strong>What would you say to a developer or enterprise just starting mobile development? What resources would you share?<\/strong><\/p>\n<p><em>YL<\/em>: We weighed our options, including investigating platform specific development languages and tools. We wanted to make sure we were doing the right thing for our users, and looking back (and forward), I&#8217;m very glad that we took this road. If we had to do it again, we&#8217;d make the same decision to go with Xamarin. My advice to new developers? If you have to deliver apps for multiple platforms \u2013 as the majority of developers do \u2013 select a toolset that allows you to create a truly native design and to share as much code as possible. Doing this, you won\u2019t end up with multiple products with completely different codebases that become increasingly difficult to test and scale.<\/p>\n<p>Xamarin has high quality documentation, especially for developers new to mobile development. Mobile platform development documentation tends to be very dry, but Xamarin provides many easy-to-understand tutorials and samples. Personally, I most appreciate that Xamarin tutorials take time to explain platform concepts that might be foreign to new developers. I\u2019d encourage any developer to use the <a href=\"https:\/\/developer.xamarin.com\/guides\/\" target=\"_blank\">Xamarin Guide<\/a> for well-written, relevant, and easily searchable topics.<\/p>\n<h2>Learn More and Get Started with Xamarin for Visual Studio<\/h2>\n<p>Pix was recently named one of the 50 Best Apps of the Year by Time magazine! You can find the full list <a href=\"http:\/\/time.com\/4549647\/best-apps-year-2016\/?iid=sr-link1\">here<\/a>.<\/p>\n<p>To learn how our customers around the world are building amazing apps, visit <a href=\"http:\/\/xamarin.com\/customers\" target=\"_blank\">xamarin.com\/customers<\/a>, and start building your own today at <a href=\"http:\/\/xamarin.com\/download\" target=\"_blank\">xamarin.com\/download<\/a>.\t\t<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Your browser does not support the video tag. Microsoft Pix turns iPhone users into better photographers, using artificial intelligence and machine learning to automatically capture the best shots. Even before a user presses the shutter button, Microsoft&#8217;s rich computational photography algorithms make key adjustments, including: optimizing for lighting, recognizing and focusing on faces, and automatically [&hellip;]<\/p>\n","protected":false},"author":564,"featured_media":28508,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2],"tags":[9],"class_list":["post-28493","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-developers","tag-case-studies"],"acf":[],"blog_post_summary":"<p>Your browser does not support the video tag. Microsoft Pix turns iPhone users into better photographers, using artificial intelligence and machine learning to automatically capture the best shots. Even before a user presses the shutter button, Microsoft&#8217;s rich computational photography algorithms make key adjustments, including: optimizing for lighting, recognizing and focusing on faces, and automatically [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/28493","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/users\/564"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/comments?post=28493"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/posts\/28493\/revisions"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/media?parent=28493"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/categories?post=28493"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/xamarin\/wp-json\/wp\/v2\/tags?post=28493"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}