Facial Recognition and the Intelligent Kiosk App

Developer Support

This post is provided by Senior Application Development Manager, Rich Maines, who introduces us to the Intelligent Kiosk App using facial recognition and Cognitive Services APIs.

I noticed this sign while running across the Brooklyn Bridge this past January.

“Area Under NYPD Video Surveillance”Was the system actually analyzing pedestrians in real time? Was it be able to identify expressions?

I can’t speak to this particular system and privacy implications aside, facial recognition technology, including the ability to distinguish between expressions, is advancing rapidly. Though the security sector is one of the principle drivers, facial recognition technologies are allowing companies across multiple industries opportunities to provide custom services based on your image. Besides the obvious workflows for the enterprise, such as for identity purposes (Windows Hello), there is great potential for consumer markets as well. I’m sorry, but Minority Report-style facial recognition advertising has arrived (“Hello Mr. Yakamoto and welcome back to the GAP!”). Consumer oriented services will scan faces, analyzing expressions and other information for customized marketing purposes that further humanize products and services.

I’d like to introduce you to the Intelligent Kiosk app. The Intelligent Kiosk is a Windows 10 Universal App showcasing workflows and experiences using Microsoft Cognitive Services. Microsoft Cognitive Services let you build apps with powerful algorithms using just a few lines of code. They work across devices and platforms such as iOS, Android, and Windows.. The kiosk is now one of the Cognitive Services samples and an Open Source project on GitHub. You can find the project here.

The kiosk app is a great example of an intelligent application, one that can understand human faces and predict their age, gender and emotion. It can do so from live or still images. It also allows you to quickly create and train a model to perform face identification against any pre-determined set of faces. Included demos are shown below, I’ll briefly introduce you to a few of my favorites.

Greeting Kiosk

The Greeting Kiosk is an example of a hands-free and real-time workflow for identifying when a person in front of the camera matches a known face (from your kiosk’s pre-trained model). This could be used, for example, as a simple log-in mechanism to greet people by name and present them with some personalized information, like photos, weather, news, etc. Or if you’re my son, to display a kind message to his sister upon entering his room.

Emotion API Explorer

This is a great example of the power of emotion recognition. This demo allows you to load images from Bing or Local Files, or from a live image captured from your web camera to display the top-3 emotions displayed (the Emotion API recognizes 8 emotions: neutral, happy, sad, anger, surprise, disgust, or contempt).

Automatic Photo Capture

A hands-free photo booth experience that explores age and gender prediction as well as face identification by using real-time face tracking to detect when people are near the camera. The model that detects the faces can also be trained to identify specific people. When one of those people is identified, their names will also appear next to their age. I tried to throw it off by changing my appearance from the image used for the training model by wearing a hat and different glasses. It was still able to recognize me 100% of the time with the added benefit of being slightly generous in predicting my age, I’m 47.

Realtime Crowd Insights

This real-time demo analyzes emotion, age, gender and identification while multiple people are looking at the camera. The face tracking is done on the client by Windows 10, but the rest is done by calling the APIs (on the lower-right corner of the image is the latency between each round of API calls).  At the bottom of the screen, it shows several charts with insights about the people that it has seen, like number of unique faces, demographics and a timeline graph of the average emotion response of all the faces looking at the camera, while next to each face it also displays the emotion, age, gender, unique face id and identification of that particular face. The identification is done against the set of faces added via the Face Identification Setup page.  A note on privacy: In order to count unique faces, this demo uses a feature in the Face APIs to store the faces that it sees in a FaceList, and then it compares the faces as they come in with all the faces that it has previously seen. That face list of previously seen faces is only kept for the current hour, in case the question comes up, and it is also discarded when the demo is closed. For that reason, the demographics and overall stats charts are also erased hourly (and when the demo is closed).

My kids and I had a blast with these demos, thinking of various scenarios in which our expressions can be used in the real world and for me, this is the real power of the Intelligent Kiosk app, demonstrating ‘the art of the possible.’

Premier Support for Developers provides strategic technology guidance, critical support coverage, and a range of essential services to help teams optimize development lifecycles and improve software quality.  Contact your Application Development Manager (ADM) or email us to learn more about what we can do for you.

Developer Support App Dev Customer Success Account Manager, Microsoft Developer Support

Follow