Orchestrating TURN Servers for Cloud Deployment

Anastasia Zolochevska

Background

We recently collaborated with Aveva, an industrial design and management software company. The goal for Aveva was to improve its remote rendering service and extend this technology to support an ever-growing list of platforms and devices including HoloLens. As a result of our collaboration, we built 3DToolkit, a toolkit for creating powerful cloud-based 3D experiences which stream to low-powered devices that are traditionally out of reach.

One of the key decisions we had to make was the video transport technology. After investigating different options, WebRTC emerged as the best choice for this project. WebRTC is designed to work peer-to-peer. In our case, one of the peers is a cloud server that streams video, and the other peer is a client device that might need to traverse NAT gateways and firewalls. For these cases, WebRTC APIs use STUN servers to get the IP address of the device, and TURN servers to function as relay servers. This code story will cover how we orchestrated TURN servers for cloud deployment in our work with Aveva.

The Challenge

There are free STUN servers available to use but maintaining TURN servers that relay video traffic is resource intensive. As part of our solution, we had to set up and maintain our own TURN servers. The challenge was to create a simple process for deployment and to find an implementation of TURN that would satisfy the following requirements:

1. Supports STUN server functionality
2. Compliant with base TURN and STUN specs RFC 5766, RFC 3489, RFC 5389, RFC 6062.
3. Free and open source
4. Can be easily set up on the Linux platform
5. Works over both TCP and UDP protocols
6. Supports TURN authentication mechanisms such as long-term credentials and time-limited secret-based authentication
7. Uses a database to store authentication data in a central location
8. Supports protocols TLS, DTLS with a real world certificate
9. Supports load balancing

The Solution

After investigating different implementations of TURN servers that would satisfy all of the above requirements we decided to use coturn, an open source and fully spec-compliant TURN server.

Simplest TURN server

To run the first simple version of TURN server that satisfies first 5 requirements, we deployed an Ubuntu virtual machine in Azure, added rules to open ports 3478 and 5349, connected to the machine remotely via SSH and ran the following commands:

sudo apt-get update 
sudo apt-get install -y dnsutils 
sudo apt-get install -y coturn
sudo rm -rf /var/lib/apt/lists/*

Next, we created a file with the script listed below. It prints configuration information to /etc/turnserver.conf , such as listening, relay and external IPs and a reference to a self-signed certificate. It also adds credentials to the SQLite database that is running on the same server using the turnadmin command.

internalIp="$(ip a | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1')"
externalIp="$(dig +short myip.opendns.com @resolver1.opendns.com)"

echo "listening-port=3478
tls-listening-port=5349
listening-ip="$internalIp"
relay-ip="$internalIp"
external-ip="$externalIp"
realm=$3
server-name=$3
lt-cred-mech
userdb=/var/lib/turn/turndb
# use real-valid certificate/privatekey files
cert=/etc/ssl/turn_server_cert.pem
pkey=/etc/ssl/turn_server_pkey.pem
 
no-stdout-log"  | tee /etc/turnserver.conf


turnadmin -a -u $1 -p $2 -r $3

turnserver

The script requires 3 parameters (username, password, and realm) that will be sent as authentication data to the TURN server. Running the script will start the TURN server.

sudo bash deploy.sh user password123 somerealm.com

To validate that TURN server works, we used WebRTC Trickle ICE page.

Docker container with simple TURN server

To make our server reusable and easy to deploy in any operating system and environment, we Dockerized the above scripts. This image is available in Docker Hub.

With this setup, we can run the Docker container on any Unix VM with Docker installed:

sudo docker run -d -p 3478:3478 -p 3478:3478/udp --restart=always zolochevska/turn-server username password realm

More advanced TURN server in Azure

To be able to use a TURN server in a production environment, we needed it to satisfy the last four items in our requirements list. This situation leads to a much more complicated deployment process, as it requires:

• deploying a special kind of server that would balance the load for TURN servers instead of relaying traffic
• deploying as many VMs with a Docker container as needed to properly balance the load
• setting up a PostgreSQL database to store authentication data for long-term credentials and/or time-limited secret-based authentication
• installing the certificate to every instance of the server.

We selected Azure as our hosting environment for the set of load-balanced TURN servers and have used a number of its services such as Ubuntu VM, Azure Database for PostreSQL, Azure KeyVault, and Azure Virtual Network. To automate the process of deployment to Azure, we developed scripts and ARM templates that make it possible to set up load-balanced TURN servers ready for production within 10 minutes.

The project’s GitHub page contains steps for deployment on either a Unix or Windows working environment. The main step is automated deployment via ARM template of x number of VMs and running the specified Docker container on them.

We shared a Docker image with coturn server set up the way it works for our project on Docker Hub. If you want to make modifications, you are welcome to fork the original repo on Github, create a Docker image based on it and specify it as a parameter for your deployment template.

The following screenshot gives insight into the settings that configure deployment of TURN servers.

Conclusion

The scripts, templates and Docker images that we created made it possible to set up a production-ready, load-balanced coturn implementation of TURN server to Azure within 10 minutes. It was proven to work at scale with a massive video transfer for our use case.  At the Aveva World Summit in October 2017, we were able to demonstrate the detailed rendering of a 400 million polygon model of an oil platform from both on-premises and Azure-based rendering services streamed into a range of devices, including HoloLens.

Everything we built for this project is open-sourced and we welcome feedback in the comments below and contributions on GitHub.

Resources

• 3DToolkit main GitHub repo
• Repo with scripts, templates and instructions for deploying TURN server
• Docker image for the simple coturn server
• Docker image for coturn server that supports load balancing, secret-based authentication, and certificates

Anastasia Zolochevska

Follow