Microsoft Research Ships Intelligent Apps with the Power of C# and AI

Microsoft Pix turns iPhone users into better photographers, using artificial intelligence and machine learning to automatically capture the best shots. Even before a user presses the shutter button, Microsoft’s rich computational photography algorithms make key adjustments, including: optimizing for lighting, recognizing and focusing on faces, and automatically selecting the best shots from a “burst” of photos.

Microsoft Pix is the result of decades of computational photography research across several product and research teams. Today, we’ve invited Yung-Shin Lin, Principal Software Engineering Manager, and Josh Weisberg, Principal PM, to share how they got started with mobile development, and how Xamarin for Visual Studio and HockeyApp help them innovate at a rapid pace.

Tell us a little bit about Microsoft Research, your team, and your role. Have you always been a developer?

Josh Weisberg: Our team, the Computational Photography team, is part of Microsoft Research’s (MSR) Cognitive Services group. Our team is unique: early last year, we transitioned from researchers who build apps to showcase their research to a cross-functional Research & Development team. Our PMs, developers, and designers work closely with our researchers to create apps that deliver unique and compelling functionality with simple user interfaces. I’m actually not a developer, but I work closely with our developers, engineers, designers, and UX researchers to bring these apps to life.

Yung-Shin Lin: We are a small, but agile, team. In addition to Josh, our team is made up of one designer, five researchers, and seven developers. Personally, I’ve been a developer in the Microsoft Windows organization for 18 years – as a software engineer, development lead, and I joined Microsoft Research last year with my current role, engineering manager. I’ve enjoyed the power of .NET and C#, beginning when I worked on the first version of .NET framework APIs.

Tell us about the app and what prompted your team to build it. What were your goals?

JW: Today’s phones take pretty good pictures, but there are many situations where they don’t. We realized that there was an opportunity to utilize Artificial Intelligence (AI) and Machine Learning (ML), combined with continuous frame buffering, to teach a camera how to make better photos.

In v1, we focused on people, which we believe are the most important subjects. When Pix detects a person in a photo, AI kicks in, telling it that the person is the most important element in the photo. This begins with the capture settings, and then extends to picking out the best frame.

How does it fit into your or your department’s mobile strategy?

JW: For many years, Microsoft’s researchers have developed very sophisticated technology that enables new photo and video scenarios. We want to create apps that enable users to take advantage of this advanced technology and techniques, without needing to be computer science or photography experts.

In other words, our goal is to empower our customers, bringing Microsoft ‘s computational photography to as many users as possible. Since most consumers use their mobile phones for photography, our primary focus is these devices.

Walk us through the app – what does it allow users to do?

JW: Microsoft Pix helps you capture the best picture every time you push the button. When Machine Learning is at its best, we see the results, not the effort. You might recognize some of the capabilities we’ve brought from Microsoft Hyperlapse, Microsoft Selfie, Blink (now Live Image), and other computational photography projects.

When a user takes photo, Microsoft Pix takes a burst of photos to make sure nothing is missed, selects best images from the burst, and tweaks lighting, contrast, and focus to suggest the best option. When Microsoft Pix detects faces, it automatically adjusts exposure to help faces look better, even in difficult lighting situations. It also uses proprietary multi-frame denoise technology, which uses other burst frames to enhance the Best Image, so faces and backgrounds are as true-to-life as possible. Microsoft Pix also optimizes videos: if the app detects any motion, it creates a “Live Image” (looping video), and helps users stabilize and time lapse existing videos.

Why did your team choose Xamarin?

JW: We wanted to deliver Microsoft Pix across multiple mobile platforms, and based on our experience using Xamarin for Hyperlapse, it was an easy decision to choose Xamarin. Many of our developers have worked on other Microsoft products, so they have considerable C# experience, and Xamarin helped them get started faster.

YL: We knew that we’d get a huge productivity gain from both from using our C# knowledge and sharing architecture and code across different mobile platforms.

We’re very happy about the flexibility and reusability we’ve seen with Xamarin. We create a common business logic layer that’s shared across platforms, and we’ve designed a platform-dependent layer to tailor the UI to match users’ expectations and to leverage platform-specific APIs – like photo capture and gallery. Due to the ease of use for .NET P/Invoke, we can easily “talk” to our algorithm, written in high performant C++ code, and we use this same C++ code across all platforms.

What do you think about when you hear “native app”?

YL: For me, “native” isn’t about the language or framework used to develop the app. It’s more important that the app fits the distinct personality of the operating system and performs like any other app on the platform.

For Microsoft Pix, we want a user interface that’s familiar to the users, and we want to achieve high performance, even with AI’s heavy computation needs.

How long did it take to ship the app, from design to deploy?

JW: As the PM, I developed the idea over time, starting in 2014. I completed the specs and preliminary designs in late summer 2015, and we started development in December 2015. From a coding perspective, it took about 7 months.

YL: I’m pretty amazed at how fast the team shipped the first release – it took the team just 7 months. We wrote our first line of Microsoft Pix mid-January, and, due to a combination of our team’s agility, hard work, existing C# experience, and the power of Xamarin, we got to market extremely quickly, launching the app in July of this year.

How do you use HockeyApp?

YL: HockeyApp is indispensable in our daily process. We want to continuously improve our quality and reliability for our users, and, during internal releases for dogfood users, we closely monitor HockeyApp crash reports and use it to automatically file bugs in Visual Studio. We prioritize and constantly fix the top 10 crashes – and continue to do so after we ship and gather real-world crash data. After our first few updates, we’ve drastically improved our reliability and quality.

What’s your team planning to build next?

JW: We’ve successfully shipped the iOS version of Microsoft Pix and are now actively working on the Android version. We’ve released 5 updates to the iOS version (as of September), and will continue to add new features and fix bugs.

YL: I’m very excited to work on the Android version of Microsoft Pix, using our cross-platform design and shared code. Xamarin has helped us get our iOS version to market quickly, and, with our Android version, we’ll be able to reuse much of the codebase and take advantage of cross-platform design. Since we don’t have to start from scratch or work in another different language, we hope to deliver our Android version soon.

What would you say to a developer or enterprise just starting mobile development? What resources would you share?

YL: We weighed our options, including investigating platform specific development languages and tools. We wanted to make sure we were doing the right thing for our users, and looking back (and forward), I’m very glad that we took this road. If we had to do it again, we’d make the same decision to go with Xamarin. My advice to new developers? If you have to deliver apps for multiple platforms – as the majority of developers do – select a toolset that allows you to create a truly native design and to share as much code as possible. Doing this, you won’t end up with multiple products with completely different codebases that become increasingly difficult to test and scale.

Xamarin has high quality documentation, especially for developers new to mobile development. Mobile platform development documentation tends to be very dry, but Xamarin provides many easy-to-understand tutorials and samples. Personally, I most appreciate that Xamarin tutorials take time to explain platform concepts that might be foreign to new developers. I’d encourage any developer to use the Xamarin Guide for well-written, relevant, and easily searchable topics.

Learn More and Get Started with Xamarin for Visual Studio

Pix was recently named one of the 50 Best Apps of the Year by Time magazine! You can find the full list here.

To learn how our customers around the world are building amazing apps, visit xamarin.com/customers, and start building your own today at xamarin.com/download.