{"id":30895,"date":"2020-12-07T14:00:22","date_gmt":"2020-12-07T21:00:22","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/dotnet\/?p=30895"},"modified":"2020-12-07T10:48:15","modified_gmt":"2020-12-07T17:48:15","slug":"guest-post-bring-lights-in-unity-into-the-real-world-using-philips-hue","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/dotnet\/guest-post-bring-lights-in-unity-into-the-real-world-using-philips-hue\/","title":{"rendered":"[Guest Post] Bring lights in Unity into the real-world using Philips Hue"},"content":{"rendered":"

The article was written by Paul Marsh, a long time Microsoft Developer from Microsoft BASIC on an 8-bit Dragon 32 through to Azure and now Unity 3D. He has worked with a number of enterprise businesses, retail and social media companies. Currently he is in a joint owner of a UK based simulation\/game company with his wife.\u00a0<\/em><\/p>\n

A game or simulation can influence the real world through other mediums than just screens and speakers, such as using dedicated haptic \u201crumble pads\u201d controllers. I wanted to extend that influence to include altering the colors in the player\u2019s environment. What if we could synchronize the light cast on the player\u2019s character with actual lights in the player\u2019s room? A search light casting its beam across their face, or onboard an alien infested space craft with the murky emergency lights filling their room? With my willing assistant, Terrance, a .NET NuGet package, a couple of Philips Hue lights, and Unity 3D, let\u2019s see if it can be achieved.<\/p>\n

\"Terrance<\/p>\n

Spoiler alert \u2013 if you want to see a quick test of the results, then watch my Unity Dev Log 6a – Physical Light Teaser<\/a><\/p>\n

Philips Hue Play Bars<\/h2>\n

\"Philips \"Philips<\/p>\n

There are numerous controllable lights but for this demonstration I am using the Philips Hue Play Bars. These are LED lights that can emit a range of colors. They are controlled by a Philips Bridge, which in turn is programmable via a REST API<\/a> (note, you have to sign up to the API). Since this is .NET, there’s likely to be a NuGet package out there to make, ahem, \u201clight work\u201d out of using this API. For the demo, I\u2019m using the open-source Q42.HueApi<\/a> NuGet package.<\/p>\n

Creating the demo<\/h2>\n

The first step is to create a new Unity project and set the Project Settings<\/strong> > Player<\/strong> to .NET Standard.<\/p>\n

\"Setting<\/p>\n

Adding the Q42 NuGet Package to Unity<\/h2>\n

Currently, adding NuGet packages into Unity requires a more manual approach than perhaps you are used to. You might find a better approach, but I created a small .NET console project and added the package. Then, you can take the two managed .NET Framework 4.x DLLs, Q42.HueApi<\/code> and Q42.HueApi.ColorConverters<\/code> and place them into the Unity Project under the Plugins folder.\n\"adding<\/p>\n

Now, you have everything ready to code against.<\/p>\n

Controlling the lights<\/h2>\n

This section is a basic introduction to controlling the Philips lights.<\/p>\n

The initial handshake<\/h3>\n

The very first thing you need to do is get the Application Key (App Key) from the bridge. This requires you to supply a couple of facts:<\/p>\n

    \n
  1. Application Name \u2013 the name of your Unity app. It doesn\u2019t have to be accurate. It’s just a way to differentiate it with your Philips Bridge.<\/li>\n
  2. Device Name \u2013 the name of the device the App is running on. Again, just needs to be unique to your Bridge.<\/li>\n<\/ol>\n

    Getting the Bridges<\/h2>\n

    To get an App Key from the Bridge you now need to discover the Bridge like in the following example:<\/p>\n

    public async Task RegisterAppWithHueBridge()\r\n{\r\n    IBridgeLocator locator = new HttpBridgeLocator();\r\n    var timeout = TimeSpan.FromSeconds(5);\r\n    var bridges = await locator.LocateBridgesAsync(timeout);\r\n\r\n    \/\/ Assuming we have only one bridge\r\n    var bridge = bridges.First();\r\n    string ipAddressOfTheBridge = bridge.IpAddress;\r\n    var client = new LocalHueClient(ipAddressOfTheBridge);\r\n\r\n    \/\/ Get the key\r\n    var appKey = await client.RegisterAsync(\r\n        hueSettings.AppName, \r\n        hueSettings.DeviceName); \r\n}\r\n<\/code><\/pre>\n
    Note the comment, this requires physically pressing the accept connection button on your Philips Bridge, that is, it can take some time to walk to the device and back again. If you want to use this in a real application, then you’ll need to provide a nice \u2018waiting\u2019 UI.\nThe good thing is that you only need to go through the process once. Once you have the key you can hang on to it, so you will want to store that somewhere. I serialize it to disk by setting the property of a custom made HueSettings MonoBehaviour that resides in the game hierarchy. For example:<\/p>\n
    public class HueSettings : MonoBehaviour\r\n{\r\n    [SerializeField]\r\n    string appKey;\r\n\r\n    [SerializeField]\r\n    string appName;\r\n\r\n    [SerializeField]\r\n    string deviceName;\r\n\r\n    public string AppKey { get => appKey; set => appKey = value; }\r\n\r\n    public string AppName { get => appName; set => appName = value; }\r\n\r\n    public string DeviceName { get => deviceName; set => deviceName = value; }\r\n}\r\n<\/code><\/pre>\n
    Getting the Lights<\/h2>\n
    Once you’ve connected to the located Bridge, you can initialize the client with the App Key and discover the available lights connected to that Bridge.<\/p>\n
    this.client = new LocalHueClient(ipAddressOfTheBridge);\r\n\r\n\r\nif (!string.IsNullOrEmpty(hueSettings.AppKey))\r\n{\r\n    client.Initialize(hueSettings.AppKey);\r\n}\r\n\r\nthis.lights = await client.GetLightsAsync();\r\n\r\n<\/code><\/pre>\n
    Setting the light color<\/h2>\n
    Almost there, now how to control the lights\u2026<\/p>\n
    public async Task ChangeLight(string lightName, UnityEngine.Color color)\r\n{\r\n    if (client == null)\r\n    {\r\n        return;\r\n    }\r\n\r\n    var lightToChange = lights.FirstOrDefault((l) => l.Name == lightName);\r\n    if (lightToChange != null)\r\n    {\r\n        var command = new LightCommand();\r\n        var lightColor = new RGBColor(color.r, color.g, color.b);\r\n        command.TurnOn().SetColor(lightColor);\r\n\r\n        var lightsToAlter = new string[] { lightToChange.Id };\r\n        await client.SendCommandAsync(command, lightsToAlter);\r\n    }\r\n}\r\n<\/code><\/pre>\n
    Each light is identified by a name configured in the Philips App. You can either discover the names from the returned lights enumerable or just supply the known names. Whichever way you choose, once you have a light object you can Send a Command to it or to several lights at the same time. In the previous example, a command is created to Turn On the light (doesn\u2019t matter if it\u2019s already on) and then Set the Color of the light. Careful though, you must convert from a Unity color to a Philips color via the RGBColor class.\nOne last thing to remember is to turn off the lights when your app closes. You could do this from the OnDestroy()<\/code> or OnApplicationQuit()<\/code> Unity methods. One trick is to send a Command to all the lights by not supplying any target lights.<\/p>\n
    public async Task TurnOff()\r\n{\r\n    if (client != null)\r\n    {\r\n        var command = new LightCommand();\r\n        command.TurnOff();\r\n        await client.SendCommandAsync(command);\r\n    }\r\n}\r\n<\/code><\/pre>\n
    Now that you have control over the lights, let\u2019s do something with them.<\/p>\n
    Capturing in-game light information<\/h2>\n
    The problem \u2013 capturing the total light on a surface, not just single rays but multiple light sources, reflections, and so on.\n\"a<\/p>\n
    In my scene, I have a single spotlight source directed at the side of a character\u2019s face. What I want to do is to match the Philips light located to the right of the player to the same color. We could just grab the color the light is set to and use that. That\u2019s okay but as you\u2019ll see next, it\u2019s not very accurate.<\/p>\n
    \"a<\/p>\n
    Now, you have multiple light sources on the characters face and the resulting color is a combination of them. In fact, it\u2019s a combination of all sorts of light emitters. The lighting of the face consists of multiple light sources, reflections, ambient light, shadows, etc. Also, objects can affect the light before it reaches the character\u2019s face. For example, a window blind.<\/p>\n
    \"reducing<\/p>\n
    This means you need a way to examine the light on the character\u2019s face rather than simply aggregate the light emitters.<\/p>\n
    Capture the light via a Camera<\/h2>\n
    The solution I’m using is to place a dedicated camera near to the character\u2019s face. Its only job is to capture the face, therefore its Viewport and Clipping Planes have been constrained to only capture the side of the face.<\/p>\n
    \"camera<\/p>\n
    The real magic behind this is that each camera can render its results to a Target Texture.\n\"Camera<\/p>\n
    By referencing the same texture in code, you can analyze the colors.<\/p>\n
    this.colourCamera = GetComponent<Camera>();\r\nthis.renderTexture = this.colourCamera.targetTexture;\r\n<\/code><\/pre>\n
    In Unity, you can use a Coroutine to implement a long-polling color analysis and light setting method.<\/p>\n
    private IEnumerator FindAndSendAverageColor()\r\n{\r\n    while (!isCalculatingFaceAverage)\r\n    { \r\n        \/\/ create a copy of the texture\r\n        Texture2D tex2d = new Texture2D(renderTexture.width,\r\n                                renderTexture.height,\r\n                                TextureFormat.RGB24, false);\r\n        RenderTexture.active = renderTexture;\r\n        tex2d.ReadPixels(new Rect(0, 0, \r\n                                renderTexture.width,\r\n                                renderTexture.height),\r\n                                0, 0);\r\n        tex2d.Apply();\r\n\r\n        \/\/ get all the colors\r\n        var detectorX = renderTexture.width;\r\n        var detectorY = renderTexture.height;\r\n        var colours = tex2d.GetPixels(0, 0, \r\n            renderTexture.width, renderTexture.height);\r\n\r\n\r\n        var averageColor = AverageWeightedColor(colours);\r\n\r\n        \/\/ set the light and breath or ignore and breath\r\n        if (averageColor.r + averageColor.g + averageColor.b > 0)\r\n        {\r\n\r\n            hueLightHelper.ChangeLight(hueLightName, this.averageColor)\r\n                .ConfigureAwait(false);\r\n\r\n            yield return new WaitForSeconds(0.2f);\r\n        }\r\n        else\r\n        {\r\n            yield return new WaitForSeconds(0.5f);\r\n        }\r\n    }\r\n}\r\n<\/code><\/pre>\n
    The Camera is rendering what it can see to the texture. You then calculate the average colors, using an algorithm of our choice, and set your chosen Philips light to the result. In this demo, I used a very simple average with a little twist to say the resulting colors must total something with enough color\/light (colorGate) to make it interesting, that is, ignore deep shadows.<\/p>\n
    private Color AverageWeightedColor(Color[] colors) \r\n{\r\n    var total = 0;\r\n    var r = 0f; var g = 0f; var b = 0f;\r\n    for (var i = 0; i< colors.Length; i++) \r\n    {\r\n        if (colors[i].r + colors[i].g + colors[i].b > colorGate)\r\n        {\r\n            r += colors[i].r > colorGate ? colors[i].r : 0f;\r\n            g += colors[i].g > colorGate ? colors[i].g : 0f;\r\n            b += colors[i].b > colorGate ? colors[i].b : 0f;\r\n            total++;\r\n        }\r\n    }\r\n    return new Color(r\/total, g\/total, b\/total, 1);\r\n}\r\n<\/code><\/pre>\n
    You can now capture the light cast onto a game object, the character in this case, and emit a corresponding color to a light in the real world.<\/p>\n
    \"Example<\/p>\n
    Or however many lights you want to use. My video camera NB struggles to capture the actual light color. Honestly, it\u2019s much closer in real life.\n\"Example<\/p>\n
    Summary<\/h2>\n
    One of the major advantages of Unity is that it is able to utilize libraries from the wider .NET ecosystem allowing the game developer to, literally, bring other tools and hardware into play. In this post we have utilized a .NET library for Philips Hue to control the lighting, capture light information in Unity, and then apply the colors to Hue lights in the real world. I hope you enjoy an emmersive time playing with Unity and Philips Hue.<\/p>\n
    You can find a YouTube video version of this and more links at Unity Dev Log 6a – Physical Light Teaser<\/a> and Unity Dev Log 6b – Implementing Physical Lights.<\/a><\/p>\n
    A version of the scripts used can be found at the paulio\/UnityPhilipsLights<\/a> repository on GitHub.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Bring lights in Unity into the real world using Philips Hue.<\/p>\n","protected":false},"author":47463,"featured_media":30896,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[685,756,7184],"tags":[4,7178,7214,139],"class_list":["post-30895","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dotnet","category-csharp","category-game-development","tag-net","tag-games","tag-phillips-hue","tag-unity"],"acf":[],"blog_post_summary":"
    Bring lights in Unity into the real world using Philips Hue.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/30895","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/users\/47463"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/comments?post=30895"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/posts\/30895\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media\/30896"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/media?parent=30895"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/categories?post=30895"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/dotnet\/wp-json\/wp\/v2\/tags?post=30895"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}