Big Data Archives

Observability for Event Stream Processing with Azure Functions, Event Hubs, and Application Insights

Shervyna Ruan May 13, 2021 May 13, 2021 05/13/21

With distributed tracing handled by Azure Functions, Azure Event Hubs, and Azure Application Insights behind the scenes, Azure Application Insights provides useful monitoring visualizations of the system that helped us easily understand system performances and troubleshoot failures.

Assessing the Severity of Acne via Cell Phone Selfie Images Using A Deep Learning Model

Hang Zhang February 5, 2019 Feb 5, 2019 02/5/19

Background Watch this episode of the Decoded Show for an in-depth look with Gary Bradski, the inventor of OpenCV, and the Nestlé team. In the Global Burden of Disease Study 2015, Acne was identified as the 8th most common disease in the world and is estimated to affect as many as 700 million people globally. This skin health ...

Running Parallel Apache Spark Notebook Workloads On Azure Databricks

Clemens Wolff January 18, 2019 Jan 18, 2019 01/18/19

Background and Problem Definition In today’s fast-moving world, having access to up-to-date business metrics is key to making data-driven customer-centric decisions. With over 1000 daily flights servicing more than 100 cities and 42 million customers per year, JetBlue has a lot of data to crunch, answering questions such as: What is the ...

Real-Time Time Series Analysis at Scale for Trending Topics Detection

Omri Mendels January 2, 2019 Jan 2, 2019 01/2/19

Managing big cities and providing citizens public services requires municipalities to have a keen understanding of what citizens care the most about. And, as the cities we live in seek more and more to become the 'smart cities' of tomorrow, this means gathering and analyzing vast amounts of data. Data gathered from social and municipal sources...

Social Stream Pipeline on Databricks with auto-scaling and CI/CD using Travis

Mor Shemesh December 12, 2018 Dec 12, 2018 12/12/18

Background For the tech companies designing tomorrow’s smart cities, making local authorities able to collect and analyze large quantities of data from many different sources and mediums is critical. Data can come from different sources – from posts on social media and data automatically collected from IoT devices, to information ...

Unsupervised driver safety estimation at scale, a collaboration with Pointer Telocation

Omri Mendels July 30, 2018 Jul 30, 2018 07/30/18

Background Connected cars, which have the ability to collect and transfer data via cellular networks, offer us the opportunity to make road transport safer, more organized and more ecologically friendly. To push forward widespread adoption and help the technology fully enter the mainstream, manufacturers and third parties, like insurance ...

Runtime Configuration of Spark Streaming Jobs

Kevin Hartman May 1, 2018 May 1, 2018 05/1/18

Background Project Fortis is a social data ingestion, analysis, and visualization platform. Originally developed by Microsoft in collaboration with the United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA), Fortis provides planners and scientists with tools to gain insight from social media, public websites and ...

Azure Event Hub Ingestion at Scale with Python and Kubernetes

Tomer Rosenthal January 24, 2018 Jan 24, 2018 01/24/18

Background The increasing availability of networked vehicles is creating new transformative opportunities for the automotive industry. With the right tools to safeguard privacy and security of users, the massive amount of data generated by connected cars can be leveraged to create groundbreaking new services such as remote onboard ...

IoT Sports Sensor Machine Learning Helps Amateurs Up Their Game

Patty Ryan June 29, 2017 Jun 29, 2017 06/29/17

Recently, Microsoft partnered with the Professional Ski Instructors of America and the American Association of Snowboard Instructors, (PSIA-AASI) to use wearable IoT sensors in order to develop a machine learning model of skiing skills and expertise levels. PSIA-AASI continually evaluates new technology and methods for nurturing learning ...

Project Fortis: Accelerating UN Humanitarian Aid Planning with GraphQL

Erik Schlegel May 10, 2017 May 10, 2017 05/10/17

Background The United Nations Office for the Coordination of Humanitarian Affairs (OCHA) is in charge of planning responses to emergencies and humanitarian crises around the world, between dozens of organizations. Their activities include establishing camps for refugees and reacting to events like epidemics, famines and terrorist attacks. ...

CSE Developer Blog

Big Data Archives | CSE Developer Blog