We used a private Docker registry to deploy the Docker images for service deployment. Private registries are useful when you don\u2019t want to expose the internals of your service infrastructure as open source images on the Docker Hub. Although Docker Hub hosts private registries for a monthly fee, it is also easy to host your own private registry on premesis or on other cloud platforms such as Microsoft Azure. These registries can hold images of all your deployment services, from support services like HTTP servers to application-level images for things like websites and background workers.<\/p>\n
The private registry server is deployed from the Docker Hub and its image store is backed by Azure blob storage. This makes our Docker registry \u2018portable\u2019 allowing us to deploy services using Fleet by pushing images to our private repository from anywhere with access to Azure storage, including your local machine:<\/p>\n
![\"Architecture\"](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2015-07-21-TokTvDoceboFleetCaseStudy_images-image001.png\")
<\/p>\n
Figure 1. Service Deployment via Azure Storage-backed Private Docker Registry<\/p>\n
In the image above, we use our no-cost single Docker Hub repository to store the Docker image of our registry server. The server image is built with our Azure storage credentials and can be run from anywhere with an Internet connection.<\/p>\n
This enables the architecture in Figure 1<\/strong>, which allows us to deploy our registry as a Fleet global service, which basically means one registry server runs on every server. This is because Docker requires images in private repositories be tagged with the name of their host. By hosting a light-weight registry server on each CoreOS instance we greatly simplify deploying images.<\/p>\n Once our registry service is deployed across the cluster with Fleet, we can deploy any image within that registry as a service on the cluster. The graphic above depicts the scenario of a developer, running a registry on their local machine and pushing an image to Azure storage via the server. The service is then restarted and the new image is deployed to the appropriate nodes.<\/p>\n Using an external image store allows us to run a private Docker image repository without having to worry about SSL and authenticating users to our repository. We also get the advantage of the redundancy and backup features of Azure Storage.<\/p>\n Service Discovery<\/strong><\/p>\n CoreOS uses etcd<\/u>, a demon service that is essentially a distributed key-value store that allows services to announce their presence by publishing keys with a Time-to-live (TTL). Each deployed application instance is deployed with an accompanying \u2018announcer\u2019 service which is simply a bash script that periodically writes its host ip address and port number for it\u2019s corresponding web server.<\/p>\n Figure 2. Service Discovery with \u2018Sidekick\u2019 Services & Etcd<\/p>\n Fleet allows us to specifically define announcer or \u2018sidekick<\/a>\u2019 services, which are guaranteed to run on the same machine as the service it monitors. When the accompanying application service goes down so does the sidekick service. Because etcd has time-to-live values for each key etcd keeps up-to-date in the case of application failures, updates and restarts since the sidekick will stop writing to etcd.<\/p>\n Figure 2<\/strong> shows each router<\/strong> instance subscribes to etcd and it uses published keys in etcd to build its routing and load balancing rules via confd<\/a> templating. For example, App1<\/strong> exists on the first two VMs and between the two instances their corresponding \u2018sidekick\u2019 services update the service directory key App1 with the key-value pairs @1: 192.168.1.1:3000 and @2 192.168.1.2:3000. Routing and load-balancing within the cluster is done via Nginx<\/a>, a popular high performance web server. Because the nginx<\/strong> router is subscribed to etcd it automatically rebuilds its routing template. Requests going for app1<\/strong> will route to either of these two machines in a round-robin load-balancing manner. The same goes for App2 – this design allows for any number of applications to be deployed backed by any number of instances.<\/p>\n Here\u2019s what the App1 service<\/a> file<\/a> looks like:<\/p>\n This Unit File<\/a> refers to the docker image example-app in our private azure docker registry. To start this unit we simply do:<\/p>\n The X-Fleet Conflicts tag in the unit file instructs fleet that we don\u2019t want more than one of this unit, an instance of the service, running on the same machine in order to have high availability.<\/p>\n Routing & Load Balancing<\/strong><\/p>\n The entire cluster sits behind a load balancer and has one public virtual IP address. This public IP address points to an Azure load balancer, which serves requests to any of our nodes.<\/p>\n One advantage of using an external load balancer for the cluster is that you only need one public IP address, rather than a unique one for each instance. Cloud service providers such as Microsoft Azure have limits to public IP allocation per subscription. Also, as machines go up and down, the public IP address doesn\u2019t change and simplifies DNS zone file settings.<\/p>\n Figure 3. Azure Load Balancer spreads loads to each internal NginX instance<\/p>\n When a user request comes in, because we have the router<\/strong> running and listening on port 80 and port 443 on each node, we can handle the request no matter what node it comes to. Further, because of etcd<\/strong> service discovery each router has knowledge about where all the services are located and can route the request the appropriate container.<\/p>\n Figure 4. Internal NginX instance routes request and load balances between app containers<\/p>\n This means that the actual container, which provides the service, doesn\u2019t necessarily need to be on the machine that the Azure load balancer selects, as show in Figure 5.<\/strong><\/p>\n Figure 5. The Machines Selected by Azure Load Balancer doesn\u2019t need to actually run requested App<\/p>\n Furthermore, the Azure load balancer load balances requests amongst our router, but each router<\/strong> service unit load balances the containers for each service.<\/p>\n This prototype solution we built during the hackfest will be the basis of Tok.tv and Docebo\u2019s microservice architecture.<\/p>\n How to Deploy High Availability Apps to Azure using CoreOS with Fleet & etcd \u2013 A getting started guide & code sample for Docker Orchestration w\/ Fleet<\/a><\/p>\n Azure Storage driver for Docker Registry Server<\/a><\/p>\n![\"Architecture\"](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2015-07-21-TokTvDoceboFleetCaseStudy_images-image002.png\")
<\/p>\n# example-app@.service\r\n# the '@' in the file name denotes that this unit file is a model Description=Example High-Availabilty Web App After=router.service \r\n\r\n[Service]\r\nEnvironmentFile=\/etc\/environment\r\nExecStartPre=\/usr\/bin\/docker pull localhost:5000\/example-app:latest\r\nExecStart=\/usr\/bin\/docker run --name example -p 3000:3000 localhost:5000\/example-app:latest ExecStop=\/usr\/bin\/docker stop example\r\nExecStopPost=\/usr\/bin\/docker kill example\r\nExecStopPost=\/usr\/bin\/docker rm example\r\n\r\n[X-Fleet]\r\nConflicts=example-app@*.service\r\n<\/code><\/pre>\n<\/div>\n# upload the service model to fleet\r\nfleetctl submit example-app@.service\r\n# start an instance of the service\r\nfleetctl start example-app@1.service\r\n<\/code><\/pre>\n<\/div>\n![\"Architecture\"](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2015-07-21-TokTvDoceboFleetCaseStudy_images-image003.png\")
<\/p>\n![\"Architecture\"](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2015-07-21-TokTvDoceboFleetCaseStudy_images-image004.png\")
<\/p>\n![\"Architecture\"](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2020\/03\/2015-07-21-TokTvDoceboFleetCaseStudy_images-image005.png\")
<\/p>\nCode Artifacts<\/h2>\n
Opportunities for Reuse<\/h2>\n