{"id":2878,"date":"2017-04-12T20:03:50","date_gmt":"2017-04-12T20:03:50","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=2878"},"modified":"2020-03-23T16:13:29","modified_gmt":"2020-03-23T23:13:29","slug":"reproducible-data-science-analysis-experimental-data-pachyderm-azure","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/reproducible-data-science-analysis-experimental-data-pachyderm-azure\/","title":{"rendered":"Reproducible Data Science – Analysis of Experimental Data with Pachyderm and Azure"},"content":{"rendered":"

Inside a nondescript building in a Vancouver industrial park, General Fusion<\/a> is creating something that could forever change how the world produces energy. Since its founding in 2002, the Canadian cleantech company has risen to the forefront of the race to harness fusion energy which has the potential to supply the world with almost limitless, clean, safe, and on-demand energy. General Fusion\u2019s approach, called magnetized target fusion, involves injecting a ring of superheated hydrogen gas (plasma) into an enclosure, then quickly collapsing the enclosure. The compression of the plasma ignites a reaction that produces a large amount of energy with only helium as a byproduct.<\/p>\n

General Fusion operates several plasma injectors which superheat hydrogen gas to millions of degrees, creating a \u201ccompact toroid plasma\u201d \u2014 a magnetized ring of ionized gas which lasts for a few milliseconds and is the fuel for a fusion energy power plant. A single plasma injector fires hundreds of such shots in a day, each shot generating gigabytes of data from sensors measuring things like temperature, density, and magnetic field strength. Through an iterative process, plasma physicists at General Fusion analyze this data and adjust their experiments accordingly with the goal of creating the world\u2019s first fusion power plant.<\/p>\n

With over one hundred thousand plasma shots spanning seven years, their total data set is approaching 100TB in size. General Fusion\u2019s existing on-premise data infrastructure was limiting their ability to effectively analyze all of this data. To ensure that full scientific use could be made of this data, they were in need of a data platform that could:<\/p>\n

    \n
  1. Enable big data analytics across thousands of experiments and terabytes of data<\/li>\n
  2. Reliably produce data and analysis when sensor calibrations or algorithms are updated<\/li>\n
  3. Preserve different versions of results for different versions of calibrations or algorithmns<\/li>\n
  4. Enable collaboration with other plasma physicists and the sharing of scientific progress<\/li>\n
  5. Scale as the quantity of data increases<\/li>\n<\/ol>\n

    Engineers at General Fusion were quickly drawn to a data platform that met all their requirements: Pachyderm<\/a>. Pachyderm is an open source data analytics platform that is deployed on top of Kubernetes<\/a>, a container orchestration framework. We partnered with General Fusion to develop and deploy their new Pachyderm-based data infrastructure to Microsoft Azure. This post walks through General Fusion\u2019s new data architecture and how we deployed it to Azure.<\/p>\n

    Background<\/h2>\n

    Pachyderm offers a modern alternative to Hadoop and other distributed data processing platforms by using containers as its core processing primitive. In comparison to Hadoop where MapReduce jobs are specified as Java classes, Pachyderm users create containers to process the data. As a result, users can employ and combine any tools, languages, or libraries (e.g. R, Python, OpenCV, CNTK, etc.) to process their data. Pachyderm will take care of injecting data into the container and parallelizing the workload by replicating the container, giving each instance a different set of the data to process.<\/p>\n

    Pachyderm also provides provenance and version control of data, allowing users to view diffs and promote collaboration with other consumers of the data. With provenance, data is tracked through all transformations and analyses. This tracking enables data to be traced as it travels through a dataflow and allows for a dataflow to be replayed with its original inputs to each processing step.<\/p>\n

    Pachyderm itself is built on top of Kubernetes and deploys all of the data processing containers within Kubernetes. Originally developed by Google and hosted by the Cloud Native Computing Foundation<\/a>, Kubernetes is an open sourced platform for managing containerized applications. Kubernetes deploys any number of container replicas across your node cluster and takes care of, among other things, replication, auto-scaling, load balancing, and service discovery.<\/p>\n

    Requirements<\/h2>\n

    In addition to the node resources given by the Kubernetes cluster, Pachyderm requires:<\/p>\n