{"id":3394,"date":"2017-06-29T15:49:57","date_gmt":"2017-06-29T22:49:57","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/?p=3394"},"modified":"2020-03-14T20:40:54","modified_gmt":"2020-03-15T03:40:54","slug":"iot-sports-sensor-machine-learning-helps-amateurs-up-their-game","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/iot-sports-sensor-machine-learning-helps-amateurs-up-their-game\/","title":{"rendered":"IoT Sports Sensor Machine Learning Helps Amateurs Up Their Game"},"content":{"rendered":"

Recently, Microsoft partnered with the Professional Ski Instructors of America and the American Association of Snowboard Instructors, (PSIA-AASI)<\/a>\u00a0to use wearable IoT sensors in order to develop a machine learning model of skiing skills and expertise levels.\u00a0 PSIA-AASI continually evaluates new technology and methods for nurturing learning experiences for developing skiers and snowboarders.<\/p>\n

IoT Human Sensor Data Enables Skill Measurement<\/h2>\n

Watching a professional skier practicing drills, it\u2019s easy to recognize their expertise level.\u00a0 Our challenge was to create data to help characterize this expertise level.\u00a0 With wearable IoT sensors, we can collect positional and motion data that allow us to measure this expertise level distinction between professionals and amateurs with high precision and accuracy.\u00a0 In our analysis, we discovered the sensor data from just nine body positions provides ample signal to generate distinct activity signatures for the professional skiers\u00a0when compared with the amateurs.<\/p>\n

\"Image
PSIA-AASI Members Generate Skiing Sensor Data at Snowbird, Utah<\/figcaption><\/figure><\/p>\n

Examining the sensor data, one can see the pros\u2019 tight adherence to proper form throughout the skill, lack of erratic movement, and precision in execution. These distinct differences in absolute and relative measures of these nine body positions allow us to construct a powerful and simple classification model to categorize skiers into different expertise levels.<\/p>\n

We think this type of classification model can be used by the amateur to help them understand differences between their performance and that of the pro, and allow them to improve overall form and skill execution. \u00a0Ski and snowboard instructors can customize training strategies for each trainee, based on the insights provided by the model and the quantitative data analysis.\u00a0\u00a0As a result,\u00a0training can become more efficient.\u00a0 Over time, and with more data, more models can be created to differentiate finer-grained expertise and skill execution levels.<\/p>\n

Data Gathering to Sports Activity Machine Learning<\/h2>\n

With PSIA-AASI, we wanted to allow amateurs to compare their own skiing data to the pros’ and classify their skill level, as well as to examine specific positional and gestural differences in their skill performance.\u00a0\u00a0The Microsoft and PSIA-AASI teams worked\u00a0together at the\u00a0Snowbird ski resort to gather the field data and build the concrete data model that would give aspiring amateurs guidance on how to improve.\u00a0\u00a0In this code story, we’ll describe the steps we took to gather the data and develop the model.\u00a0 We’ll also provide links to the R script and data set that you can use to recreate this solution.<\/p>\n

\"Image
Nick Herrin of PSIA-AASI with Steve Guggenheimer of Microsoft<\/figcaption><\/figure><\/p>\n

<\/p>\n

IoT Device and Raw Sensor Data<\/h3>\n

Each of the wearable IoT sensors measures position, acceleration, and rotation individually, and records it with a time stamp.\u00a0 The variables emitted include positional variables x, y and z that represent position in three-dimensional space<\/a>; rotational matrix variables qW, qX, qY, qZ that represent 3D rotation<\/a>; and aX, aY and aZ that represent coordinate\u00a0acceleration<\/a>.<\/p>\n

While the sample rate of sensors varies, we found 100 Hz to be a minimum rate for modeling.\u00a0 There are a number of IoT sensors that capture these data at this sample rate (more details\u00a0on hardware\u00a0options<\/a>).<\/p>\n

Below is an example of the raw data variables emitted by each one of the sensors.<\/p>\n

<\/p>\n

Human Sensor Placement<\/h3>\n

We placed the sensors according to the diagram included below.\u00a0 The data from the bright green sensors\u00a0were used in the final classification model, as\u00a0features generated from these sensors’ data were highly predictive of skier expertise.\u00a0 The\u00a0features derived from the light green colored sensors’ data\u00a0were not as predictive and were discarded from the final model.<\/p>\n

We calibrated the sensors to align the timestamp for each sampling before gathering data.\u00a0\u00a0If you cannot calibrate sensors prior to\u00a0data gathering, this manual timestamp alignment can be\u00a0done as a post-processing step, presuming the sampling rate was high enough, and there is minimal missing signal.\u00a0 For reference, here is a time align R script example<\/a>.<\/p>\n

\"Sensor<\/p>\n

Experimental Design<\/h3>\n

We\u00a0set out to capture\u00a0data for several PSIA-AASI professional-level skiers and several intermediate skiers as they performed a number of skill drills.\u00a0 Our skill drills included short, medium and large radius turns.\u00a0 We noted the start of the actual drill in order to exclude non-drill data from the data model.\u00a0 We also noted the name, drill and skill level of the skier in order to annotate the data file later.<\/p>\n

\"Image
Making Field Notes on Experiment<\/figcaption><\/figure><\/p>\n

We stored our experimental data locally, then batch uploaded it to an Azure SQL database\u00a0after the experiments were complete.\u00a0 We also tested devices using Bluetooth to stream the data to a mobile device carried by the skier, though in modeling we only used the locally stored data as we didn’t have a time constraint requiring real-time streaming, and the\u00a0locally stored data had no data loss.<\/p>\n

Clean and Transform the Data<\/h3>\n

We wrote our data to Azure SQL, and applied experiment and athlete label data, then imported it into the Azure Machine Learning<\/a> workspace for the rest of the data transformation and modeling.\u00a0 Options for AML data import include Azure SQL Database, Azure Blob Storage, Azure Table, Azure Document DB and others.<\/p>\n

\"Time
Time Series Visualization of x, y, and z Positional Variables<\/figcaption><\/figure><\/p>\n

Originally, the data was saved in a “long” format, meaning that every row in the dataset represents readings from a sensor at one sampling time stamp.\u00a0 You can see the time series\u00a0graph of each of these rows above. \u00a0If we had five sensors, there would be five rows of records for each sampling time stamp.\u00a0 This data format presents some challenges for data analysis and modeling.\u00a0 We transformed this raw “long” data into the “wide” data format, with one row for each athlete and experiment. Refer to the R script<\/a> for the specific code we used for this transformation, as well as the transformations and modeling that follow.<\/p>\n

Generate Virtual Sensory Data from Physical Sensors<\/h3>\n

With guidance from the PSIA-AASI professionals, we characterized what hallmark differences might be captured between the pros and amateurs in the sensor data. These differences included the relative position of their upper body vs. lower body, the relative position of each of their legs and feet, and how they took a turn. With this domain expert insight,\u00a0we generate virtual\u00a0sensory data from the raw physical sensors\u00a0that characterize limb position relative to one another, and the upper body relative to the lower body.<\/p>\n

Later on, at the data analysis and modeling stage,\u00a0we learned\u00a0the features that best illustrate these differences in skiing include the normalized difference between upper body and lower body, the relational positions and rotation of upper body and lower body, and the relational position and rotation of the limbs.\u00a0 The screenshot below shows a few of these engineered features.\u00a0 You can find all the\u00a0extra virtual sensory data we created\u00a0in the R script<\/a>.<\/p>\n

\"R
R Script Module to Generate Virtual Sensory Data Features in Azure ML<\/figcaption><\/figure><\/p>\n

Engineer Features by Generating Periodic Descriptive Measures<\/h3>\n

Next, we broke our ski activity sample into small activity interval slices; in our case, time slices of two seconds apiece. We generated summary statistics as well as frequency and frequency covariance statistics over these time windows. We then ran a fast-discrete Fourier transform function <\/a>on each of the raw sensory signals to\u00a0convert the data from being represented in a time domain to being represented in a frequency domain. This is measured by the magnitude of each frequency (from 0Hz to half of the sample frequency, 240Hz) for a given time interval slice. These generated frequency components of our sensor signal include constant power (frequency 0), low-band average power (1-40Hz), mid-band average power (41-80Hz), and high-band average power (81-120Hz). Finally, we generated cross-correlation measures on select variables to measure the similarity of various two series combinations (for more details, see the R script <\/a>of this feature engineering).<\/p>\n

Create a Classification ML Model<\/h3>\n

After feature engineering, we had 589 observations and 2144 features.\u00a0 Training a machine learning model with such a wide data set with so few observations would have introduced over-fitting problems, even using the simplest logistic regression model.\u00a0 So, we first used ‘Filter Based Feature Selection’ in Azure ML to select the top thirty features, based on strength of their relationship with the target variable, ‘SkillLevel’, as measured by Mutual Information. \u00a0 We did the feature selection only on the training data, which is a randomly selected 70% of the original dataset.\u00a0 The remaining 30% of records were held out from the feature selection and model training to be used as validation data.<\/p>\n

\"Overview
Overview of Classification Model in Azure ML<\/figcaption><\/figure><\/p>\n

When we tested the model on the validation data, we found the model correctly predicted classification of professional vs. amateur expertise levels across all skills 99% of the time.<\/p>\n

\"Image
Model Evaluation Results with ROC Curve and Confusion Matrix in Azure ML<\/figcaption><\/figure><\/p>\n

Access the R Script and the Full Experiment<\/h3>\n

The entire R script can be accessed on GitHub<\/a>. \u00a0The end-to-end pipeline, including the data transformations, feature engineering, feature selection and machine learning model training and validation has been implemented in an Azure Machine Learning experiment and published in the Cortana Intelligence Suite Gallery<\/a>.\u00a0 With this gallery experiment, you can reproduce the results presented here.<\/p>\n

Animate Sensor Data with Avatars<\/h3>\n

For even more insight, you can turn your sensor data into your own action avatar and compare it with avatars from the pros’ sensor data.<\/p>\n

We processed the data into a 3D skeletal animation format and imported the data into the Unity development environment<\/a> to create the video you see below.\u00a0 We relied on the Humanoid Animation Retarget system to create a visual representation of a skier drill.\u00a0 In addition to this video, the 3D animation can be visualized from different perspectives and angles to better understand differences of the amateur on the left\u00a0and one of the\u00a0professionals on the right\u00a0in terms of body positioning.<\/p>\n