{"id":3667,"date":"2017-06-28T19:39:42","date_gmt":"2017-06-28T19:39:42","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=3667"},"modified":"2020-03-14T21:00:43","modified_gmt":"2020-03-15T04:00:43","slug":"revisiting-interop-unity-uwp-directx","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/revisiting-interop-unity-uwp-directx\/","title":{"rendered":"Revisiting InterOp with Unity, UWP and DirectX"},"content":{"rendered":"

In the last few years \u2014\u00a0particularly with the release of new 3D-focused devices in Virtual Reality (VR), New User Input, and Mixed Reality \u2014 we’ve seen Unity <\/a>grow from an easy platform for game developers into one of the main development tools for graphical 2D\/3D, simulation, visualization and cross-platform software solutions.<\/p>\n

Unity has its own runtime system and, while it makes it easier to create heavy graphics and UI-focused applications from games to simulations without having to build common components from scratch, the runtime can, in some cases, become limiting. In our hackfest with Aveva<\/a>, we encountered a requirement that made us consider a hybrid approach in which we would use the workstation and server GPU while still use Unity Runtime as the main platform for handling device input, 3D model processing, camera, physics, and\u00a0UI. To use this approach, we needed to build plugins and utilize the WebRTC library.<\/p>\n

Fortunately, Unity supports low-level plugins that allow access to native APIs to get closer to the hardware. On the Microsoft Windows platform, native code plugins can be written for Standalone (typically PC Desktop) x86\/x64 and Universal Windows Platform (UWP)<\/strong>\u00a0x86\/x64\u00a0to access DirectX for graphics and device APIs for hardware features. Unity provides documentation for low-level plugins for DirectX, \u00a0but limited details are available about the full interaction from the plugin back to Unity. This effort brings us back to C\/C++ and InterOp<\/strong> when .NET and managed code were introduced, and the need to utilize the wealth of existing native libraries.<\/p>\n

In our\u00a0reference example (UnityWithWebRTC)<\/a>, the requirements are pushed further with the need to use a UWP package coming from Nuget.org<\/a>. WebRTC<\/a> is an open source project that provides browsers, mobile applications, and apps real-time communication (RTC) capabilities using simple APIs. The UWP Sample App\u00a0PeerCC\u00a0<\/a>was used as the reference for creating the bridging library between Unity and the WebRTC package.<\/p>\n

There are multiple ways to handle the integration. The example uses this setup:<\/p>\n

\"Image<\/p>\n

Low-Level\u00a0Plugin<\/h4>\n

The image frame is handled in the DirectX plugin, which takes an H.264 encoded byte array for decoding and conversion to the matching texture color format and updating the texture object. Processing the image frame in the Unity script in a CoRoutine<\/a> would not work, as the operations would be limited to the\u00a0script update cycle. The plugin access requires DllImport to call the native code from the managed Unity script (ControlScript.cs<\/a>).<\/p>\n

[DllImport(\"TexturesUWP\")]\r\nprivate static extern void SetTextureFromUnity(System.IntPtr texture, int w, int h);\r\n\r\n[DllImport(\"TexturesUWP\")]\r\nprivate static extern void ProcessRawFrame(uint w, uint h, IntPtr yPlane, uint yStride, IntPtr uPlane, uint uStride,\r\n    IntPtr vPlane, uint vStride);\r\n\r\n[DllImport(\"TexturesUWP\")]\r\nprivate static extern void ProcessH264Frame(uint w, uint h, IntPtr data, uint dataSize);\r\n\r\n[DllImport(\"TexturesUWP\")]\r\nprivate static extern IntPtr GetRenderEventFunc();\r\n\r\n[DllImport(\"TexturesUWP\")]\r\nprivate static extern void SetPluginMode(int mode);<\/pre>\n
The DllImport attribute provides a link back to the available method\/functions in the native library to be called from the managed C# code in the Unity Runtime. Now that the code in the native library can be executed, the next challenge is passing parameters. Passing by Value is not an issue as the name implies: the value is merely passed to the method. It becomes tricky, however, when Pass by Reference is needed, such as in a use case where a byte array needs to be passed from managed code to unmanaged\/native methods.<\/p>\n
private void EncodedVideo_OnEncodedVideoFrame(uint w, uint h, byte[] data)\r\n{\r\n    frameCounter++;\r\n    fpsCounter++;\r\n\r\n    messageText = data.Length.ToString();\r\n\r\n    if (data.Length == 0)\r\n        return;\r\n\r\n    if (frame_ready_receive)\r\n        frame_ready_receive = false;\r\n    else\r\n        return;\r\n\r\n    GCHandle buf = GCHandle.Alloc(data, GCHandleType.Pinned);\r\n    ProcessH264Frame(w, h, buf.AddrOfPinnedObject(), (uint)data.Length);\r\n    buf.Free();\r\n}<\/pre>\n
GCHandle.Alloc is used with GCHandleType.Pinned since we are passing by reference a byte array to a native plugin to prevent garbage collection and the object from being moved during the plugin execution.<\/p>\n
Managing Updates<\/strong><\/p>\n
The Unity Runtime and DirectX Plugin share a Texture Object. The texture provides the image data to a\u00a0Quad GameObject in the Unity Scene while the Texture data is updated in the plugin.<\/p>\n
private void CreateTextureAndPassToPlugin()\r\n{\r\n#if !UNITY_EDITOR\r\n    RenderTexture.transform.localScale = new Vector3(-TextureScale, (float) textureHeight \/ textureWidth * TextureScale, 1f);\r\n\r\n    Texture2D tex = new Texture2D(textureWidth, textureHeight, TextureFormat.ARGB32, false);        \r\n    tex.filterMode = FilterMode.Point;       \r\n    tex.Apply();\r\n    RenderTexture.material.mainTexture = tex;\r\n    SetTextureFromUnity(tex.GetNativeTexturePtr(), tex.width, tex.height);\r\n#endif\r\n}<\/pre>\n
The plugin is signaled from the Unity Runtime when it can execute an update on the shared graphics context objects via a CoRoutine <\/a>that waits for EndOfFrame and then invokes GL.IssuePluginEvent(GetRenderEventFunc(), 1).<\/p>\n
private IEnumerator CallPluginAtEndOfFrames()\r\n{\r\n    while (true)\r\n    {\r\n        \/\/ Wait until all frame rendering is done\r\n        yield return new WaitForEndOfFrame();\r\n\r\n        \/\/ Issue a plugin event with arbitrary integer identifier.\r\n        \/\/ The plugin can distinguish between different\r\n        \/\/ things it needs to do based on this ID.\r\n        \/\/ For our simple plugin, it does not matter which ID we pass here.\r\n\r\n#if !UNITY_EDITOR\r\n\r\n        switch (PluginMode)\r\n        {\r\n            case 0:\r\n                if (!frame_ready_receive)\r\n                {\r\n                    GL.IssuePluginEvent(GetRenderEventFunc(), 1);\r\n                    frame_ready_receive = true;\r\n                }\r\n                break;\r\n            default:\r\n                GL.IssuePluginEvent(GetRenderEventFunc(), 1);\r\n                break;                \r\n        }          \r\n#endif\r\n    }\r\n}<\/pre>\n
On the plugin DLL, the GetRenderEventFunc() triggers when the plugin can invoke the graphics context\u00a0UpdateSubresource(d3dtex, 0, NULL, dataPtr, rowPitch, 0)<\/strong> to update the Texture object. Since the texture is defined by Unity, the update here carries forward to the texture and object in the Unity Scene:<\/p>\n
static void UNITY_INTERFACE_API OnRenderEvent(int eventID)\r\n{\r\n    switch (PluginMode)\r\n    {\r\n    case 0:     \/\/ WebRTC Image Frame Rendering\r\n        ProcessARGBFrameData();\r\n        break;\r\n    case 1:     \/\/ Invoke\r\n        ProcessTestFrameData();\r\n        break;\r\n    case 2:     \/\/ Invoke\r\n        if(isARGBFrameReady)\r\n        {\r\n            UpdateUnityTexture(g_TextureHandle, g_TextureWidth * pixelSize, argbDataBuf);\r\n            isARGBFrameReady = false;\r\n        }\r\n        break;\r\n    }\r\n}\r\n\r\nextern \"C\" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API GetRenderEventFunc()\r\n{\r\n    return OnRenderEvent;\r\n}<\/pre>\n
void UpdateUnityTexture(void* textureHandle, int rowPitch, void* dataPtr)\r\n{\r\n    ID3D11Texture2D* d3dtex = (ID3D11Texture2D*)textureHandle;\r\n    assert(d3dtex);\r\n\r\n    ID3D11DeviceContext* ctx = NULL;\r\n    m_Device->GetImmediateContext(&ctx);\r\n\r\n    ctx->UpdateSubresource(d3dtex, 0, NULL, dataPtr, rowPitch, 0);\r\n    ctx->Release();\r\n}<\/pre>\n
UWP Libraries<\/h3>\n
Compared to low-level plugins, working with Managed UWP Libraries is simpler \u2014\u00a0the main step involves putting the .DLL or .WINMD into the Project Hierarchy and defining its properties.<\/p>\n
 \"Image<\/p>\n
The UWP WebRTC libraries can also be included in the plugins and invoked from Unity C# Scripts which is really managed code running on .NET Runtime when deployed on Windows. WebRtc.DLL and WebRtc.WINMD are the only files needed and the “Don’t Process” (no patching) checkbox is set.<\/p>\n
 \"Image<\/p>\n
With UWP Plugins, the libraries can be referenced in the Unity Scripts the same way as referenced libraries or dependencies. The code sections typically need to be wrapped by compiler directives as they are not recognized by the Unity Editor.<\/p>\n
#if !UNITY_EDITOR\r\nusing Org.WebRtc;\r\nusing WebRtcWrapper;\r\nusing PeerConnectionClient.Model;\r\nusing PeerConnectionClient.Signalling;\r\nusing PeerConnectionClient.Utilities;\r\n#endif<\/pre>\n
Summary<\/h2>\n
As Unity applications become more complex, the need arises to extend Unity to provide additional capabilities. In our work with Aveva, we used Unity’s plugin architecture to utilize the workstation and server GPU, while still leveraging the Unity runtime for input and display.<\/p>\n
The Unity with WebRTC Project<\/a> is posted on GitHub to serve as a reference for the setup and interaction of components. Feel free to send us questions, comments, and feedback.<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
As Unity applications become more complex, the need arises to extend Unity to provide additional capabilities. In our work with Aveva, we used Unity’s plugin architecture to utilize the workstation and server GPU, while still leveraging the Unity runtime for input and display.<\/p>\n","protected":false},"author":21379,"featured_media":10936,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[17],"tags":[153,213,368,370],"class_list":["post-3667","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-frameworks","tag-directx","tag-interop","tag-unity","tag-universal-windows-platform-uwp"],"acf":[],"blog_post_summary":"
As Unity applications become more complex, the need arises to extend Unity to provide additional capabilities. In our work with Aveva, we used Unity’s plugin architecture to utilize the workstation and server GPU, while still leveraging the Unity runtime for input and display.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/3667","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21379"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=3667"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/3667\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/10936"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=3667"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=3667"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=3667"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}