We recently worked with Situm<\/a>, a company that offers high precision indoor navigation. Situm’s platform is able to ascertain the position of a\u00a0user inside a building by relying on a smartphone\u2019s sensors: WiFi, Bluetooth, magnetometer, gyroscope, and the accelerometer. Their platform is now being used in a variety of scenarios, including to provide patients with turn-by-turn navigation inside hospitals and to optimize response times of commercial security services\u00a0by tracking the locations of their security personnel. As Situm continues to grow, they wanted to provide high availability and scalability so they looked to Kubernetes on Azure. We also worked with the Azure engineering team to add support to Kubernetes for UDP workloads on Azure.<\/p>\n The Situm app running on a\u00a0smartphone continually sends sensor information to Situm’s services. To optimize for performance, this communication is done over UDP. The received sensor data is processed by artificial intelligence models to compute the positioning of the smartphone within a building. Each model tracks a user\u2019s prior movements to predict their next one. As a result, each location request from the smartphone needs to hit the service with the same artificial intelligence model loaded, otherwise there will\u00a0be a performance penalty for reloading the entire user state.<\/p>\n Kubernetes<\/a> can easily scale out Situm’s\u00a0application to handle the increased workload. In order to direct smartphone traffic to the same backend, we need to enable\u00a0load balancing with session affinity.<\/p>\n The easiest method to provision a Kubernetes cluster on Azure is through\u00a0Azure Container Service<\/a>. The underlying engine of Azure Container Service, acs-engine<\/a>, is open-source and available on GitHub, and provides a means to further customize a\u00a0Kubernetes deployment.<\/p>\n While working with Situm, we used acs-engine to deploy our cluster as we needed a private build of hyperkube<\/a>. hyperkube is a minimal container that contains all\u00a0core Kubernetes services (e.g., kubelet, apiserver, etc.)\u00a0compiled into a single binary. We needed a private build of hyperkube as the upstream change to the Kubernetes project to add support for UDP traffic to\u00a0the Azure provider had not yet been included in a release. Support for UDP traffic through the Azure load balancer is now included in Kubernetes v1.6.5 and later. For reference, here is the\u00a0pull request on the Kubernetes project to add the support<\/a>\u00a0and the subsequent pull request to backport it to v1.6.5<\/a>.<\/p>\n With a Kubernetes Deployment<\/span>\u00a0Situm can easily scale-up\/down their workload. Below is a Deployment<\/span>\u00a0manifest for a sample UDP workload listening on port 10001 (note that the protocol is explicitly set to UDP):<\/p>\n kube-proxy<\/span>\u00a0is a daemon that runs on each node and acts as a network proxy and load balancer for services on that node. kube-proxy<\/span>\u00a0watches the Kubernetes master and for each Service<\/span>\u00a0 it installs and maintains iptables<\/span>\u00a0rules which capture traffic to the Service<\/span>\u00a0and redirects traffic to a pod in the backend pool.<\/p>\n In order to expose the\u00a0Deployment<\/span>\u00a0to the outside world, we create a\u00a0\u00a0Service<\/span>\u00a0with type LoadBalancer<\/span>. Again, note that UDP is explicitly set as the protocol.<\/p>\n The default load balancing mode of a Kubernetes\u00a0Service<\/span>\u00a0is round robin, and this is implemented by Kubernetes as a chain of iptables<\/span>\u00a0forwarding rules. We can take a peek at the iptables<\/span>\u00a0 rules by ssh-ing into any of the agents:<\/p>\n The above excerpt shows a chain of forwarding rules; each rule uses the statistic module in random mode to match incoming packets. In order to attain session affinity, deploy the Kubernetes\u00a0Service<\/span>\u00a0 with\u00a0service.spec.sessionAffinity<\/span>\u00a0set to ClientIP<\/span>:<\/p>\n Once the new manifest has been applied, Kubernetes will add the following new iptables<\/span>\u00a0rules on the agents:<\/p>\n The rules use the recent module to track the source addresses of the packets. The first packet from a given IP address will make it past these rules and will be resolved by the statistic-based rules we saw earlier. For the next 10800 seconds, subsequent packets from the same source address will then match the above rule and follow the same resolution (i.e., be forwarded to the same backend pod).\u00a0<\/span><\/p>\n The solution outlined in this code story is adaptable to any workload which requires session persistence. As of Kubernetes v1.6.5 (release notes<\/a>), Kubernetes on Azure supports both UDP and TCP workloads, and respects the\u00a0Service spec’s sessionAffinity<\/span>.<\/p>\nThe Problem<\/h2>\n
The Solution<\/h2>\n
Kubernetes<\/h3>\n
apiVersion: extensions\/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n name: udp-server-deployment\r\nspec:\r\n replicas: 5\r\n template:\r\n metadata:\r\n labels:\r\n name: udp-server\r\n spec:\r\n containers:\r\n - name: udp-server\r\n image: jpoon\/udp-server\r\n imagePullPolicy: Always\r\n ports:\r\n - containerPort: 10001\r\n protocol: UDP<\/pre>\n
Session Affinity<\/h3>\n
apiVersion: v1\r\nkind: Service\r\nmetadata:\r\n name: udp-server-service\r\n labels:\r\n app: udp-server\r\nspec:\r\n type: LoadBalancer\r\n ports:\r\n - port: 10001\r\n protocol: UDP\r\n selector:\r\n name: udp-server<\/pre>\n
$ sudo iptables-save -t nat\r\n...\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m statistic --mode random --probability 0.20000000019 -j KUBE-SEP-PXZRUUO6ETLHNSK5\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-7BYVUHZG6JCWRBXE\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-TQ6Y3YOHBBRO6ZEY\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-L63KMNPKEWP6ZS3R\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -j KUBE-SEP-IGJNOFR3CZKDK3MQ\r\n...<\/pre>\n
apiVersion: v1\r\nkind: Service\r\nmetadata:\r\n name: udp-server-service\r\n labels:\r\n app: udp-server\r\nspec:\r\n type: LoadBalancer\r\n sessionAffinity: ClientIP\r\n ports:\r\n - port: 10001\r\n protocol: UDP\r\n selector:\r\n name: udp-server<\/pre>\n
$ sudo iptables-save -t nat\r\n...\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-PXZRUUO6ETLHNSK5 --mask 255.255.255.255 --rsource -j KUBE-SEP-PXZRUUO6ETLHNSK5\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-7BYVUHZG6JCWRBXE --mask 255.255.255.255 --rsource -j KUBE-SEP-7BYVUHZG6JCWRBXE\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-TQ6Y3YOHBBRO6ZEY --mask 255.255.255.255 --rsource -j KUBE-SEP-TQ6Y3YOHBBRO6ZEY\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-L63KMNPKEWP6ZS3R --mask 255.255.255.255 --rsource -j KUBE-SEP-L63KMNPKEWP6ZS3R\r\n-A KUBE-SVC-47RAPJ3AUKGKU6DC -m comment --comment \"default\/udp-server-service:\" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-IGJNOFR3CZKDK3MQ --mask 255.255.255.255 --rsource -j KUBE-SEP-IGJNOFR3CZKDK3MQ\r\n...<\/pre>\n
Opportunities for Reuse<\/h2>\n