Big Data Archives | CSE Developer Blog

Assessing the Severity of Acne via Cell Phone Selfie Images Using A Deep Learning Model

Hang ZhangFebruary 5, 2019Feb 5, 201902/5/19

Background Watch this episode of the Decoded Show for an in-depth look with Gary Bradski, the inventor of OpenCV, and the Nestlé team. In the Global Burden of Disease Study 2015, Acne was identified as the 8th most common disease in the world and is estimated to affect as many as 700 million people globally. This skin health ...

Running Parallel Apache Spark Notebook Workloads On Azure Databricks

Clemens WolffJanuary 18, 2019Jan 18, 201901/18/19

Background and Problem Definition In today’s fast-moving world, having access to up-to-date business metrics is key to making data-driven customer-centric decisions. With over 1000 daily flights servicing more than 100 cities and 42 million customers per year, JetBlue has a lot of data to crunch, answering questions such as: What is the ...

Real-time time series analysis at scale for trending topics detection

Omri MendelsJanuary 2, 2019Jan 2, 201901/2/19

Managing big cities and providing citizens public services requires municipalities to have a keen understanding of what citizens care the most about. And, as the cities we live in seek more and more to become the 'smart cities' of tomorrow, this means gathering and analyzing vast amounts of data. Data gathered from social and municipal sources...

Social Stream Pipeline on Databricks with auto-scaling and CI/CD using Travis

Mor ShemeshDecember 12, 2018Dec 12, 201812/12/18

Background For the tech companies designing tomorrow’s smart cities, making local authorities able to collect and analyze large quantities of data from many different sources and mediums is critical. Data can come from different sources – from posts on social media and data automatically collected from IoT devices, to information ...

Unsupervised driver safety estimation at scale, a collaboration with Pointer Telocation

Omri MendelsJuly 30, 2018Jul 30, 201807/30/18

Background Connected cars, which have the ability to collect and transfer data via cellular networks, offer us the opportunity to make road transport safer, more organized and more ecologically friendly. To push forward widespread adoption and help the technology fully enter the mainstream, manufacturers and third parties, like insurance ...

Runtime Configuration of Spark Streaming Jobs

Kevin HartmanMay 1, 2018May 1, 201805/1/18

Background Project Fortis is a social data ingestion, analysis, and visualization platform. Originally developed by Microsoft in collaboration with the United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA), Fortis provides planners and scientists with tools to gain insight from social media, public websites and ...

Azure Event Hub Ingestion at Scale with Python and Kubernetes

Tomer RosenthalJanuary 24, 2018Jan 24, 201801/24/18

Background The increasing availability of networked vehicles is creating new transformative opportunities for the automotive industry. With the right tools to safeguard privacy and security of users, the massive amount of data generated by connected cars can be leveraged to create groundbreaking new services such as remote onboard ...

IoT Sports Sensor Machine Learning Helps Amateurs Up Their Game

Patty RyanJune 29, 2017Jun 29, 201706/29/17

Recently, Microsoft partnered with the Professional Ski Instructors of America and the American Association of Snowboard Instructors, (PSIA-AASI) to use wearable IoT sensors in order to develop a machine learning model of skiing skills and expertise levels. PSIA-AASI continually evaluates new technology and methods for nurturing learning ...

Project Fortis: Accelerating UN Humanitarian Aid Planning with GraphQL

Erik SchlegelMay 10, 2017May 10, 201705/10/17

Background The United Nations Office for the Coordination of Humanitarian Affairs (OCHA) is in charge of planning responses to emergencies and humanitarian crises around the world, between dozens of organizations. Their activities include establishing camps for refugees and reacting to events like epidemics, famines and terrorist attacks. ...

Reproducible Data Science – Analysis of Experimental Data with Pachyderm and Azure

Jason PoonApril 12, 2017Apr 12, 201704/12/17

Inside a nondescript building in a Vancouver industrial park, General Fusion is creating something that could forever change how the world produces energy. Since its founding in 2002, the Canadian cleantech company has risen to the forefront of the race to harness fusion energy which has the potential to supply the world with almost limitless...

CSE Developer Blog