CSE Developer Blog

Running Parallel Apache Spark Notebook Workloads On Azure Databricks

January 18, 2019 Jan 18, 2019 01/18/19

Clemens Wolff

This article walks through the development of a technique for running Spark jobs in parallel on Azure Databricks. The technique enabled us to reduce the processing times for JetBlue's reporting threefold while keeping the business logic implementation straight forward. The technique can be re-used for any notebooks-based Spark workload on Azure Databricks.

Developing and Deploying a Churn Prediction Model with Azure Machine Learning Services

January 10, 2019 Jan 10, 2019 01/10/19

Patty Ryan

Deep learning algorithms capable of learning and predicting customer behavior are allowing businesses to intervene with the right retention offers at the right time. CSE recently partnered with Majid Al Futtaim Ventures (MAF) to design and deploy a machine learning solution to predict attrition.

Real-Time Time Series Analysis at Scale for Trending Topics Detection

January 2, 2019 Jan 2, 2019 01/2/19

Omri Mendels

This code story describes a collaboration with ZenCity around detecting trending topics at scale. We discuss the datasets, data preparation, models used and the deployment story for this scenario.

Improving Safety and Efficiency in BMW Manufacturing Plants with an Open Source Platform for Managing Inventory Delivery

December 19, 2018 Dec 19, 2018 12/19/18

Ryan Pedersen

Over the course of twelve months Microsoft and BMW partnered three different times to help BMW with its vision for technical transformation. An open-source package called ROS-Industrial was used to help provide the building blocks for the robotics work.

Social Stream Pipeline on Databricks with auto-scaling and CI/CD using Travis

December 12, 2018 Dec 12, 2018 12/12/18

Mor Shemesh

This code story describes CSE's work with ZenCity to create a data pipeline on Azure Databricks supported by a CI/CD pipeline on TravisCI. The aim of the collaboration was to create a pipeline capable of processing a stream of social posts, analyzing them, and identifying trends.

Active Learning for Object Detection in Partnership with Conservation Metrics

November 6, 2018 Nov 6, 2018 11/6/18

Olga Liakhovich

CSE teamed up with Conservation Metrics to use Active Learning to allow for more efficient data labeling for object detection projects.

How to Build A K8S Http API For Helm, and Serve Micro-services Using A Single IP

October 25, 2018 Oct 25, 2018 10/25/18

Itye Richter

The Commercial Software Engineering team (CSE) partnered with Axonize to automate the process of deploying apps to Kubernetes, and expose these apps to the internet via a single IP. This post is about enabling applications in your Kubernetes cluster to programmatically install helm charts and expose them through a single public facing IP.

Attaching and Detaching an Edge Node From a HDInsight Spark Cluster when running Dataiku Data Science Studio (DSS)

August 20, 2018 Aug 20, 2018 08/20/18

Amy Boyd

Earlier this year, Dataiku and Microsoft joined forces to add extra flexibility to DSS on HDInsight, and also to allow Dataiku customers to attach a persistent edge node on an HDInsight cluster – something which was previously not a feature supported by the most recent edition of Azure HDInsight.

Infrastructure as Code – On demand GPU clusters with Terraform & Jenkins

August 9, 2018 Aug 9, 2018 08/9/18

Tomer Rosenthal

Developing robust algorithms for self-driving cars requires sourcing event data from over 10 billion hours of recorded driving time. CSE worked with Cognata, a startup developing simulation platforms for autonomous vehicles, to build a Jenkins pipeline and Terraform solution that enabled our partner to dynamically scale GPU resources for their simulations.

Unsupervised driver safety estimation at scale, a collaboration with Pointer Telocation

July 30, 2018 Jul 30, 2018 07/30/18