{"id":7749,"date":"2018-06-01T09:28:34","date_gmt":"2018-06-01T16:28:34","guid":{"rendered":"\/developerblog\/?p=7749"},"modified":"2020-03-19T13:28:12","modified_gmt":"2020-03-19T20:28:12","slug":"creating-private-ethereum-consortium-kubernetes","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/creating-private-ethereum-consortium-kubernetes\/","title":{"rendered":"Building a Private Ethereum Consortium"},"content":{"rendered":"

Background<\/h2>\n

The travel industry estimates 3-5% of bookings are disputed due to discrepancies in the data held by each party and can take months to resolve.\u00a0Over the past two years, Microsoft and Webjet <\/a>have collaborated to build a blockchain-based solution, Rezchain, that helps travel companies resolve data and avoid invoice reconciliation issues. You can read more about our previous work in the following code stories: Using a Layer 7 proxy for Ethereum Blockchain<\/a>, Using Helm to Deploy Blockchain to Kubernetes<\/a><\/em>.<\/p>\n

Earlier this year, Webjet announced<\/a> the expansion of Rezchain<\/a>. One of the challenges we faced in our recent hackfest<\/a> with Webjet was how to connect each party’s Ethereum network to form a consortium. For a consortium, Ethereum nodes should be distributed across the network and, optionally, each participating member of Rezchain will host their own Ethereum nodes. In this code story, we\u2019ll share some of the lessons we learned in creating the Rezchain consortium. In particular, we’ll focus on how we solved the challenges involved with enabling Ethereum nodes to peer inside and across virtual networks.<\/p>\n

<\/p>\n

The Challenge<\/h2>\n

In this article<\/a> by Vitalik Buterin, co-founder of Ethereum, blockchain applications generally fall into three categories based on their access privileges and who can participate in the consensus process: public, consortium, and private.<\/p>\n\n\n\n\n\n\n
Transaction<\/em><\/strong><\/td>\nPublic<\/strong><\/td>\nConsortium<\/strong><\/td>\nPrivate<\/strong><\/td>\n<\/tr>\n
Send<\/em><\/td>\nAnyone<\/td>\nLimited<\/td>\nCentralized<\/td>\n<\/tr>\n
Approve<\/em><\/td>\nAnyone<\/td>\nLimited<\/td>\nCentralized<\/td>\n<\/tr>\n
Read<\/em><\/td>\nAnyone<\/td>\nLimited<\/td>\nLimited<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

In a public blockchain, anyone can read and send transactions to the chain and participate in the consensus process. On the other hand, with a private blockchain write permissions are centralized to a single organization and read permissions can be restricted to an arbitrary extent. Falling in the middle of the spectrum, a consortium blockchain has read and write permissions limited to a certain number of parties in the network.<\/p>\n

It is important that hotel booking transactions and any company-sensitive data remain private. Any data on the blockchain would be disclosed to all connected parties, and Webjet decided to store hash values to ensure sensitive details are not stored on the chain. As the blockchain eco-system is still evolving, Webjet has decided to run blockchain as consortium model at the moment. It also helps with efforts to maintain and support blockchain transaction nodes and miners. The Rezchain\u2019s consortium consists of Webjet subsidiary hotel distribution brands, such as Lots of Hotels and JacTravel, with each company controlling and hosting their own Ethereum nodes. The challenge is how to enable the Ethereum nodes from one network (Lots of Hotel) to connect and peer with Ethereum nodes of another (JacTravel) where these Ethereum nodes could be potentially located in various parts of the world and hosted under different network topologies and network constraints. Working in collaboration with Webjet, we arrived at a solution that will enable Ethereum nodes to connect over the public Internet and to form a blockchain consortium.<\/p>\n

The Solution<\/h2>\n

Before presenting the final solution, background information of the methods that Ethereum employs to discover and peer with other Ethereum nodes is necessary.<\/p>\n

Ethereum Peering<\/h2>\n

There are 3 different ways in which peers can be configured: bootnode, static nodes, and trusted nodes:<\/p>\n\n\n\n\n\n\n
<\/td>\nBootnode<\/strong><\/td>\nStatic Nodes\n<\/strong><\/td>\nTrusted Nodes\n<\/strong><\/td>\n<\/tr>\n
Uses Node Discovery<\/em><\/td>\nYes<\/td>\nNo<\/td>\nNo<\/td>\n<\/tr>\n
Transport Protocols Used<\/em><\/td>\nUDP, TCP<\/td>\nTCP<\/td>\nTCP<\/td>\n<\/tr>\n
Adheres to Max Peers<\/em><\/td>\nYes<\/td>\nYes<\/td>\nNo<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Bootnode<\/h3>\n

Bootnode<\/a> is a lightweight application used for the Node Discovery Protocol<\/a>. Bootnodes do not sync chain data. Using a UDP-based Kademlia<\/a>-like RPC protocol, Ethereum will connect and interrogate these bootnodes for the location of potential peers. The Ethereum Foundation maintains several bootnodes for the public Ethereum networks; the endpoints of which are hard-coded in the Geth source code<\/a>. The default list can be configured using the --bootnodes<\/code> option:<\/p>\n

geth --bootnodes [enode:\/\/pubkey1@ip1:port1,enode:\/\/pubkey2@ip2:port2,\u2026]<\/pre>\n
Given the list of potential peers, Geth will then attempt to connect to each peer over TCP to negotiate and ensure compatibility with protocol version, network IDs, and genesis blocks.<\/p>\n
Static Nodes<\/h3>\n
Static Nodes are pre-configured connections which are always maintained and re-connected on disconnects by Ethereum nodes. As static nodes will be connected to directly, no UDP discovery is required. They can be configured at runtime through the JavaScript console:<\/p>\n
$ admin.addPeer(\"enode:\/\/f4642fa65af50cfdea8fa7414a5def7bb7991478b768e296f5e4a54e8b995de102e0ceae2e826f293c481b5325f89be6d207b003382e18a8ecba66fbaf6416c0@33.4.2.1:30303\")\r\n$ admin.addPeer(\"enode:\/\/pubkey@ip:port\")<\/pre>\n
Or by supplying a “static-nodes.json\u201d file in the Geth data directory:<\/p>\n
$ cat GETH-DATA-DIR\/static-nodes.json\r\n[\r\n\"enode:\/\/f4642fa65af50cfdea8fa7414a5def7bb7991478b768e296f5e4a54e8b995de102e0ceae2e826f293c481b5325f89be6d207b003382e18a8ecba66fbaf6416c0@33.4.2.1:30303\",\r\n\"enode:\/\/pubkey@ip:port\"\r\n]<\/pre>\n
Trusted Nodes<\/h3>\n
Similar to Static Nodes, Trusted Nodes are pre-configured peers which Geth will always try to stay connected to even after the peer limit has been reached.<\/span> The peer limit is a limit to the number of peers that a Geth node can connect to; the default value is 25, but it can be configured using the --maxpeers<\/code> flag. Trusted Nodes can be configured by supplying a \u201ctrusted-nodes.json\u201d file in the Geth data directory:<\/p>\n
$ cat GETH-DATA-DIR\/trusted-nodes.json\r\n[\r\n\"enode:\/\/f4642fa65af50cfdea8fa7414a5def7bb7991478b768e296f5e4a54e8b995de102e0ceae2e826f293c481b5325f89be6d207b003382e18a8ecba66fbaf6416c0@33.4.2.1:30303\",\r\n\"enode:\/\/pubkey@ip:port\"\r\n]<\/pre>\n
Connecting the Consortium<\/h2>\n
In this section, we’ll present a generalized approach to connect the Ethereum nodes of each member to participate in the consensus process and form a consortium.<\/p>\n
Using Helm to Deploy Blockchain to Kubernetes<\/a> walks through deploying an Ethereum network to Kubernetes using Helm. In doing so, you will have an Ethereum network that looks like the following:<\/p>\n
\"Image

Private Ethereum Network<\/figcaption><\/figure><\/p>\n
In order to connect the Ethereum networks of each member, we applied a mixture of Ethereum’s peering strategies to enable the Ethereum nodes of one member’s network to peer and communicate with nodes from another network to form a consortium.<\/span><\/p>\n
<\/p>\n
\"\"
Ethereum Consortium with 2 Members<\/figcaption><\/figure><\/p>\n
As shown in the diagram above, building upon the original Helm chart<\/a>, we made a couple of modifications to enable peering across virtual networks.<\/p>\n
Within a member’s network, we used bootnode for discovery and peering of Ethereum nodes. In doing so, this enables easier scaling in\/out of Ethereum nodes as each new node will be able to discover other peers using the Ethereum Node Discovery Protocol. For high availability, we deployed several replicas of bootnode. On start-up, each bootnode registers against a registrar (bootnode-registrar) that maintains a list of active bootnodes. Prior to an Ethereum node launching, an HTTP call is made to the bootnode-registrar service to retrieve the list of bootnodes and then passing the list of bootnodes to the Ethereum node using the --bootnodes<\/code> flag. The bootnode-registrar<\/a> is available on GitHub and an accompanying Helm chart<\/a> can be used to deploy the networks for one of the consortium members.<\/p>\n
To connect a member’s Ethereum network to another member, we exposed the P2P ports of one Ethereum node (Geth-Tx-Public) to the public internet. Following the same pattern as the bootnode-registrar, the public Ethereum node registers<\/span> with a registrar (staticnode-registrar)<\/span> and subsequently retrieves a list of nodes that have registered. The list is then passed to Geth through the static-nodes.json configuration. The staticnode-registrar<\/a> is available on GitHub (note: the repository is named ‘bootnode-registrar’).<\/p>\n
Once the discovery and peering is complete, the Ethereum network will look like the following diagram:<\/p>\n
\"\"
Ethereum Consortium with 2 Members (Simplified view following discovery and peering)<\/figcaption><\/figure><\/p>\n
Summary<\/h2>\n
As companies begin to explore scenarios to leverage blockchain, consortium blockchains are becoming increasingly popular. This code story has walked through the learnings of our collaboration with Webjet in building a blockchain consortium. One of the biggest challenges we faced in our engagement was how to enable Ethereum nodes to be able to discover and peer with one another \u2013 an obstacle we overcame using a mixture of Ethereum peering techniques and custom services (bootnode-registrar and staticnode-registrar).\n<\/span><\/p>\n
In order to deploy our network in a more highly available nature, we’ve developed registrars <\/span>for the bootnode<\/a> and staticnodes<\/a> which we have made available on GitHub. Please note that the implementation of the registrars is incomplete and does not offer any level of security. If you have any questions or feedback on this code story, please reach out in the comments below or through the respective GitHub repositories.\n<\/span><\/p>\n
Resources<\/h2>\n