Big Data

Observability for Event Stream Processing with Azure Functions, Event Hubs, and Application Insights

May 13, 2021 May 13, 2021 05/13/21

Shervyna Ruan

With distributed tracing handled by Azure Functions, Azure Event Hubs, and Azure Application Insights behind the scenes, Azure Application Insights provides useful monitoring visualizations of the system that helped us easily understand system performances and troubleshoot failures.

Assessing the Severity of Acne via Cell Phone Selfie Images Using A Deep Learning Model

February 5, 2019 Feb 5, 2019 02/5/19

Hang Zhang

Nestlé Skin Health partnered with Microsoft to develop a deep learning model powered mobile app able to assess acne severity using only uploaded selfie images as a source.

Running Parallel Apache Spark Notebook Workloads On Azure Databricks

January 18, 2019 Jan 18, 2019 01/18/19

Clemens Wolff

This article walks through the development of a technique for running Spark jobs in parallel on Azure Databricks. The technique enabled us to reduce the processing times for JetBlue's reporting threefold while keeping the business logic implementation straight forward. The technique can be re-used for any notebooks-based Spark workload on Azure Databricks.

Real-Time Time Series Analysis at Scale for Trending Topics Detection

January 2, 2019 Jan 2, 2019 01/2/19

Omri Mendels

This code story describes a collaboration with ZenCity around detecting trending topics at scale. We discuss the datasets, data preparation, models used and the deployment story for this scenario.

Social Stream Pipeline on Databricks with auto-scaling and CI/CD using Travis

December 12, 2018 Dec 12, 2018 12/12/18

Mor Shemesh

This code story describes CSE's work with ZenCity to create a data pipeline on Azure Databricks supported by a CI/CD pipeline on TravisCI. The aim of the collaboration was to create a pipeline capable of processing a stream of social posts, analyzing them, and identifying trends.

Unsupervised driver safety estimation at scale, a collaboration with Pointer Telocation

July 30, 2018 Jul 30, 2018 07/30/18

Omri Mendels

A scalable unsupervised approach for driver safety estimation on Pointer Telocation's dataset

Runtime Configuration of Spark Streaming Jobs

May 1, 2018 May 1, 2018 05/1/18

Kevin Hartman

We achieved zero-downtime reconfiguration and management of the Spark Streaming job used in Project Fortis with Azure Service Bus.

Azure Event Hub Ingestion at Scale with Python and Kubernetes

January 24, 2018 Jan 24, 2018 01/24/18

Tomer Rosenthal

We created a solution to ingest Azure Event Hubs events at scale using Python and Kubernetes.

IoT Sports Sensor Machine Learning Helps Amateurs Up Their Game

June 29, 2017 Jun 29, 2017 06/29/17

Patty Ryan

We use IoT sensors to collect positional and motion data from professional and amateur skiers to classify expertise and skill level through machine learning.

Project Fortis: Accelerating UN Humanitarian Aid Planning with GraphQL

May 10, 2017 May 10, 2017 05/10/17

Erik Schlegel

Using GraphQL and Azure to create a data processing pipeline for identifying trends and providing insights about global humanitarian crises.

CSE Developer Blog

Big Data - CSE Developer Blog