In the last few years — particularly with the release of new 3D-focused devices in Virtual Reality (VR), New User Input, and Mixed Reality — we’ve seen Unity grow from an easy platform for game developers into one of the main development tools for graphical 2D/3D, simulation, visualization and cross-platform software solutions.

Unity has its own runtime system and, while it makes it easier to create heavy graphics and UI-focused applications from games to simulations without having to build common components from scratch, the runtime can, in some cases, become limiting. In our hackfest with Aveva, we encountered a requirement that made us consider a hybrid approach in which we would use the workstation and server GPU while still use Unity Runtime as the main platform for handling device input, 3D model processing, camera, physics, and UI. To use this approach, we needed to build plugins and utilize the WebRTC library.

Fortunately, Unity supports low-level plugins that allow access to native APIs to get closer to the hardware. On the Microsoft Windows platform, native code plugins can be written for Standalone (typically PC Desktop) x86/x64 and Universal Windows Platform (UWP) x86/x64 to access DirectX for graphics and device APIs for hardware features. Unity provides documentation for low-level plugins for DirectX,  but limited details are available about the full interaction from the plugin back to Unity. This effort brings us back to C/C++ and InterOp when .NET and managed code were introduced, and the need to utilize the wealth of existing native libraries.

In our reference example (UnityWithWebRTC), the requirements are pushed further with the need to use a UWP package coming from Nuget.org. WebRTC is an open source project that provides browsers, mobile applications, and apps real-time communication (RTC) capabilities using simple APIs. The UWP Sample App PeerCC was used as the reference for creating the bridging library between Unity and the WebRTC package.

There are multiple ways to handle the integration. The example uses this setup:

Low-Level Plugin

The image frame is handled in the DirectX plugin, which takes an H.264 encoded byte array for decoding and conversion to the matching texture color format and updating the texture object. Processing the image frame in the Unity script in a CoRoutine would not work, as the operations would be limited to the script update cycle. The plugin access requires DllImport to call the native code from the managed Unity script (ControlScript.cs).

[DllImport("TexturesUWP")]
private static extern void SetTextureFromUnity(System.IntPtr texture, int w, int h);

[DllImport("TexturesUWP")]
private static extern void ProcessRawFrame(uint w, uint h, IntPtr yPlane, uint yStride, IntPtr uPlane, uint uStride,
    IntPtr vPlane, uint vStride);

[DllImport("TexturesUWP")]
private static extern void ProcessH264Frame(uint w, uint h, IntPtr data, uint dataSize);

[DllImport("TexturesUWP")]
private static extern IntPtr GetRenderEventFunc();

[DllImport("TexturesUWP")]
private static extern void SetPluginMode(int mode);

The DllImport attribute provides a link back to the available method/functions in the native library to be called from the managed C# code in the Unity Runtime. Now that the code in the native library can be executed, the next challenge is passing parameters. Passing by Value is not an issue as the name implies: the value is merely passed to the method. It becomes tricky, however, when Pass by Reference is needed, such as in a use case where a byte array needs to be passed from managed code to unmanaged/native methods.

private void EncodedVideo_OnEncodedVideoFrame(uint w, uint h, byte[] data)
{
    frameCounter++;
    fpsCounter++;

    messageText = data.Length.ToString();

    if (data.Length == 0)
        return;

    if (frame_ready_receive)
        frame_ready_receive = false;
    else
        return;

    GCHandle buf = GCHandle.Alloc(data, GCHandleType.Pinned);
    ProcessH264Frame(w, h, buf.AddrOfPinnedObject(), (uint)data.Length);
    buf.Free();
}

GCHandle.Alloc is used with GCHandleType.Pinned since we are passing by reference a byte array to a native plugin to prevent garbage collection and the object from being moved during the plugin execution.

Managing Updates

The Unity Runtime and DirectX Plugin share a Texture Object. The texture provides the image data to a Quad GameObject in the Unity Scene while the Texture data is updated in the plugin.

private void CreateTextureAndPassToPlugin()
{
#if !UNITY_EDITOR
    RenderTexture.transform.localScale = new Vector3(-TextureScale, (float) textureHeight / textureWidth * TextureScale, 1f);

    Texture2D tex = new Texture2D(textureWidth, textureHeight, TextureFormat.ARGB32, false);        
    tex.filterMode = FilterMode.Point;       
    tex.Apply();
    RenderTexture.material.mainTexture = tex;
    SetTextureFromUnity(tex.GetNativeTexturePtr(), tex.width, tex.height);
#endif
}

The plugin is signaled from the Unity Runtime when it can execute an update on the shared graphics context objects via a CoRoutine that waits for EndOfFrame and then invokes GL.IssuePluginEvent(GetRenderEventFunc(), 1).

private IEnumerator CallPluginAtEndOfFrames()
{
    while (true)
    {
        // Wait until all frame rendering is done
        yield return new WaitForEndOfFrame();

        // Issue a plugin event with arbitrary integer identifier.
        // The plugin can distinguish between different
        // things it needs to do based on this ID.
        // For our simple plugin, it does not matter which ID we pass here.

#if !UNITY_EDITOR

        switch (PluginMode)
        {
            case 0:
                if (!frame_ready_receive)
                {
                    GL.IssuePluginEvent(GetRenderEventFunc(), 1);
                    frame_ready_receive = true;
                }
                break;
            default:
                GL.IssuePluginEvent(GetRenderEventFunc(), 1);
                break;                
        }          
#endif
    }
}

On the plugin DLL, the GetRenderEventFunc() triggers when the plugin can invoke the graphics context UpdateSubresource(d3dtex, 0, NULL, dataPtr, rowPitch, 0) to update the Texture object. Since the texture is defined by Unity, the update here carries forward to the texture and object in the Unity Scene:

static void UNITY_INTERFACE_API OnRenderEvent(int eventID)
{
    switch (PluginMode)
    {
    case 0:     // WebRTC Image Frame Rendering
        ProcessARGBFrameData();
        break;
    case 1:     // Invoke
        ProcessTestFrameData();
        break;
    case 2:     // Invoke
        if(isARGBFrameReady)
        {
            UpdateUnityTexture(g_TextureHandle, g_TextureWidth * pixelSize, argbDataBuf);
            isARGBFrameReady = false;
        }
        break;
    }
}

extern "C" UnityRenderingEvent UNITY_INTERFACE_EXPORT UNITY_INTERFACE_API GetRenderEventFunc()
{
    return OnRenderEvent;
}

void UpdateUnityTexture(void* textureHandle, int rowPitch, void* dataPtr)
{
    ID3D11Texture2D* d3dtex = (ID3D11Texture2D*)textureHandle;
    assert(d3dtex);

    ID3D11DeviceContext* ctx = NULL;
    m_Device->GetImmediateContext(&ctx);

    ctx->UpdateSubresource(d3dtex, 0, NULL, dataPtr, rowPitch, 0);
    ctx->Release();
}

UWP Libraries

Compared to low-level plugins, working with Managed UWP Libraries is simpler — the main step involves putting the .DLL or .WINMD into the Project Hierarchy and defining its properties.

The UWP WebRTC libraries can also be included in the plugins and invoked from Unity C# Scripts which is really managed code running on .NET Runtime when deployed on Windows. WebRtc.DLL and WebRtc.WINMD are the only files needed and the “Don’t Process” (no patching) checkbox is set.

With UWP Plugins, the libraries can be referenced in the Unity Scripts the same way as referenced libraries or dependencies. The code sections typically need to be wrapped by compiler directives as they are not recognized by the Unity Editor.

#if !UNITY_EDITOR
using Org.WebRtc;
using WebRtcWrapper;
using PeerConnectionClient.Model;
using PeerConnectionClient.Signalling;
using PeerConnectionClient.Utilities;
#endif

Summary

As Unity applications become more complex, the need arises to extend Unity to provide additional capabilities. In our work with Aveva, we used Unity’s plugin architecture to utilize the workstation and server GPU, while still leveraging the Unity runtime for input and display.

The Unity with WebRTC Project is posted on GitHub to serve as a reference for the setup and interaction of components. Feel free to send us questions, comments, and feedback.