OAuth 2.0 for Mixed Reality applications

Ben Greenier

OAuth 2.0 for Mixed Reality applications

We recently collaborated with Aveva, an industrial design and management software company. The goal for Aveva was to improve its remote rendering service and extend this technology to support an ever-growing list of platforms and devices including HoloLens. As a result of our collaboration, we built 3DToolkit, a toolkit for creating powerful cloud-based 3D experiences that can stream to any device.

One of the key decisions we had to make was how to secure these 3D experiences. After investigating different techniques, we chose the OAuth 2.0 industry-standard protocol to authenticate our users. This protocol is widely supported by authentication providers including Microsoft, Facebook, and Google. As a result, 3DToolkit allows users to authenticate using account information from any of these providers. While 3DToolkit provides resources for building many types of 3D streaming experiences, this code story focuses on a challenge we faced with authentication, when developing our HoloLens experience in partnership with Aveva. Since Aveva’s service handles confidential data, we needed users to authenticate before using it. To make this easy with Windows Mixed Reality we built a system to allow users to use a different and familiar device to authenticate to the Windows Mixed Reality session.

Understanding the use case

While using OAuth 2.0 to authenticate users isn’t new or exciting, there are new factors to consider inside Windows Mixed Reality experiences. For instance, typing can be a tough task when the user is using a virtual keyboard. Since a user needs to type their username and (often lengthy) password to authenticate, doing so in Windows Mixed Reality can be less than magical. However, with HoloLens and other mixed-reality experiences, the user sees both the real world and the virtual world. This scenario lets us think differently about what devices a user can use to type their credentials.

To overcome the pain of typing on a virtual keyboard, our team tried instructing the user to use a physical computer or mobile device to input this information. We found this to be a more usable experience and used this approach for our HoloLens experience.

Defining the behavior

To implement this flavor of OAuth 2.0 authentication, some additional work is required. We needed a way to securely pass the access token retrieved by the user on their physical device over to the Windows Mixed Reality application in the virtual world. Fortunately, we found an existing proposal for a similar approach as an IETF draft. This draft describes exactly the system we’re looking for; in fact, you may have seen it in use for tools like the Azure CLI.

For our application, we extended the draft’s proposal to more clearly define how our implementation would behave.  We documented a potential implementation using markdown and Swagger. We call this implementation OAuth 2.0 for devices, or oauth24d for short. In the documentation for this implementation, we define a REST-ful API designed to meet our needs.

Note that there are two actors involved, the Windows Mixed Reality application that starts the flow, and the client browser that completes the authentication flow on behalf of a user. The initiator polls the REST-ful API at an interval until it receives a response indicating the client browser has successfully completed the authentication flow. You can learn more about the specifics of each endpoint in the implementation documentation.

In the oauth24d documentation, we also provide a sample service as a NodeJS application. With the sample service, any passport-compatible authentication backend (like Microsoft, Google, Facebook, or others) is supported for user login. This means we can support different authentication providers using the same API and application logic.

Putting it all together

Given the implementation described above, we can now write Windows Mixed Reality application logic to facilitate requesting and retrieving user credentials. The experience we built for HoloLens in partnership with Aveva is powered by Unity and written in C#; however, any application that can communicate using HTTP can leverage the same REST-ful API and client-side patterns. Here’s an example leveraging the sample client used in our experience:

var client = new OAuth24DClient("http://example.com/new", "http://example.com/poll");

client.CodeComplete += (OAuth24DClient.CodeCompletionData codeData) =>
{
    // use http_status to ensure the codeData is valid. show user_code
    // and verification_url to the user so they can login via a browser.
    // the rest of the fields are consumed internally for polling
    //
    if (authData.http_status)
    {
        // show codeData.user_code and codeData.verifcation_url
    }
};

client.AuthenticationComplete += (OAuth24DClient.AuthCompletionData authData) =>
{
    // use http_status to ensure the authData is valid. add access_token to the
    // Authentication header of requests that require authentication! 
    //
    if (authData.http_status)
    {
        // store authData.access_token
    }
};

client.Authenticate();

In the example, we connect to the /new endpoint to create an authorization session. This provides a device-specific code used to determine if the user has authenticated yet, a user-specific code and a URL. We see this data in the CodeComplete event emitted by the OAuth24DClient instance. Our 3D experience will then present the user-specific code and direct the user to the URL to authenticate. Meanwhile, our application will connect to the /poll endpoint, passing along our device-specific code to determine if the user has authenticated yet. Once they have, we’ll receive an access token that is usable for authentication on behalf of the user. We see this data in the AuthenticationComplete event emitted by the OAuth24DClient instance. At this point, we have the access token in memory on our client, and can use it to authenticate to services on behalf of a user.

Summary

In this article we showed how we enabled a HoloLens experience developed in partnership with Aveva to authenticate a user, using a different physical device. This feature was critical to ensuring Aveva’s customer data remained secure, while still enabling visualization within Windows Mixed Reality. Overall, we found using a different physical device creates a better user experience than authenticating with a virtual keyboard. We also provided a sample implementation of a RESTful API, and a client that communicates with that API. These components can be used to authenticate users within any Windows Mixed Reality application.

The 3DToolkit leverages this approach to provide easy-to-use authentication features for developers building 3D experiences that are streamed over the network to Windows Mixed Reality applications. If you’re interested in trying out our approach of OAuth 2.0 for Devices, get in touch with us on GitHub! We’d love to hear your feedback.

Resources

• 3DToolkit
• OAuth 2.0 Documentation
• IETF Device Flow Draft
• OAuth2 For Devices Reference Documentation
• OAuth2 For Devices Server Examples
• OAuth2 For Devices Client Examples

Ben Greenier

Follow