{"id":37664,"date":"2019-08-23T03:00:30","date_gmt":"2019-08-23T10:00:30","guid":{"rendered":"http:\/\/devblogs.microsoft.com\/premier-developer\/?p=37664"},"modified":"2019-09-05T06:02:25","modified_gmt":"2019-09-05T13:02:25","slug":"permutation-implementation-with-ml-net","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/premier-developer\/permutation-implementation-with-ml-net\/","title":{"rendered":"Introduction to ML.NET with Permutation Feature Importance"},"content":{"rendered":"

In this post, Senior App Dev Manager Randy Park<\/a> continues with his series on Machine Learning with an experiment and introduction to ML.NET.<\/p>\n

\n

This is the continuation from the previous article<\/a> which demonstrated of feature permutations using Azure Machine Learning Studio solution. As mentioned in that article, if you would like to find out more on this powerful experiment, please read about permutation feature importance<\/a>. Put it simply, the Permutation Feature Importance (PFI) computes the permutation feature importance scores of feature variables given a trained model and a test dataset. As illustrated, doing it with Azure ML studio was easy as a few drag-and-drop operations. In this article, we would wonder what it would take on doing the same with ML.NET<\/a>.<\/p>\n

We will continue to use same dataset as mentioned in previous article which utilized a very popular exercise\u00a0dataset to conduct EDA<\/a> to determine factors that lead to attrition. The data contains 35 total columns as below:<\/p>\n

<\/p>\n

We will identify at least the top three or four factors that contribute to employee turnover. I often find general PFI examples on internet seemed rather too simple. I hope this experiment can relate the machine learning problem to real world scenarios as well as explaining the steps & efforts involved.<\/p>\n

Introduction to ML.NET implementation<\/h2>\n

The building of machine learning model comprises of four or five steps depending on who you ask or which machine learning framework or strategy you employ. Here with ML.NET, any complex ML.NET operations can be succinctly expressed utilizing high level IEstimator<\/strong> .NET interface along with other various transformers. These choices of names and such implementation are not any different from the popular ML frameworks, such as TensorFlow and Scikit-Learn pipelines.<\/p>\n

<\/p>\n

Figure 1: Sample of ML.NET Pipelines<\/em><\/p>\n

First, we begin with obvious by creating the MLContext and the ML.NET implementation begins.<\/p>\n

<\/p>\n

Figure 2: Step-0 \u2013 Instantiate MLContext<\/em><\/p>\n

As our ML.NET application also starts with an\u00a0MLContext<\/a>\u00a0object, keep in mind this singleton object offers many useful\u00a0catalogs<\/strong>. A catalog is a factory class for data loading and saving, transforms, trainers, and model operation components. On debugging time, when inspecting of MLContext instance, you may feel it seems to offer just handful of components and algorithms, but please note that other training algorithms are also available through extension libraries.<\/p>\n

Below is the step to load the training data to IDataView interface as well as provisioning test data.<\/p>\n

<\/p>\n

Figure 3: Step-1 \u2013 Instantiate IDataView<\/em><\/p>\n

Then, we will start featurization step by applying data transformers as below.<\/p>\n

<\/p>\n

Figure 4: Step-2 \u2013 Featurize with transformers<\/em><\/p>\n

I would like to make a clear separation of transformers responsible of featurizing of input dataset to the trainer. There are a few reasons, first, I would like to apply on other algorithms and make the comparisons. ML.Net offers so many great trainers and we will have to choose the trainer, typically based on the performance metrics and\/or utilization of system resources. Second, as most experienced data scientists agree, the success of building machine learning model is on the featurization.<\/p>\n

Continuing on, as this sample code seems very simple, and one-hot-encoding was no brainer for the text field categorization, though I spent considerable amount of time to choose a few other ways to vectorizing text fields and too numerous to mention to validate all the trials and errors. Make the long story short, once I am done with the featurization steps, I can now concentrate on which trainer to pick to fit the model to the featurized training data.<\/p>\n

Here, I chose SdcaLogisticRegression<\/a> trainer.<\/p>\n

<\/p>\n

Figure 5: Step-3 \u2013 Train<\/em><\/p>\n

Explaining of each of binary classification trainers are out-of-scope for this article, please go to this link<\/a> to find out more about each of the trainers mentioned in the code.<\/p>\n

<\/p>\n

Figure 5: Step-4 \u2013 Evaluate<\/em><\/p>\n

The ConsoleHelper class contains much useful utility methods developing and debugging machine learning models. As the model\u2019s accuracy is about 85% and the area under curve (AUC) reaches about 83%, this model should be good to use to predict if the employee may likely leave or stay given circumstances.<\/p>\n

===== Evaluating Model's accuracy with Test data =====\r\n* Accuracy: 85.66%\r\n* Area Under Curve: 82.94%\r\n* Area under Precision recall Curve: 54.56%\r\n* F1Score: 38.81%\r\n* LogLoss: .5\r\n* LogLossReduction: .23\r\n* PositivePrecision: .65\r\n* PositiveRecall: .28\r\n* NegativePrecision: .87\r\n* NegativeRecall: 97.07%\r\n************************************************************<\/pre>\n
Now the primary purpose of the article begins here, the Feature Permutation step should occur right after the evaluation of the trained model. The main purpose of evaluation is on model performance and make decision based on the metrics of the model, but you can also draw out other observations including feature permutations. As the trained model is available, using the other random-sampled test dataset, different set of permutation of features will occur under the hood and will identify which of the features have heavier importance.<\/p>\n
<\/p>\n
Figure 5: Step-4 \u2013 PFI<\/em><\/p>\n
Now most important aspect of this project is to find the \u201cwhy\u201d or \u201cwhat\u201d contributes to attrition rather than providing of the accurate model. The BinaryClassfication.PermutaitonFeatureImportance() will provide the metrics, and make sure that we provide the featurized test data plus the trained model. Since we created separate featurizePipeline transformer, we can easily create IDataView permuteTestData.<\/p>\n
What does the above PFI findings highlight? Below are the observations;<\/p>\n\n\n\n\n\n\n\n
Top 4 Features<\/th>\nComments of this Observation<\/th>\n<\/tr>\n
Overtime<\/td>\nHigher<\/strong>\u00a0reported Overtime, employees are\u00a0more<\/strong>\u00a0likely to leave.<\/td>\n<\/tr>\n
Marital Status<\/td>\nCertain <\/strong>group of marital status is more likely to leave than the other group. As far as which group, it could be identified through the debugging of each permutation slot associated.<\/td>\n<\/tr>\n
Department<\/td>\nCertain department employees are more likely leave than other departments. Again, the identification could have been done as well.<\/td>\n<\/tr>\n
Years At Company<\/td>\nNumber of years at the same company are factored. Again, is it more years more<\/strong>\u00a0likely to leave or less<\/strong> likely to leave?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
As with the first article, this exercise should not be looked at as the de-facto conclusion regarding attrition. However, it\u2019s interesting to note of top concern of \u201cOvertime\u201d agreed with R implementation of correlation coefficients and Azure Machine Learning Studio solution.<\/p>\n
Wrap Up<\/h2>\n
As mentioned in the previous article, PFI metrics are very useful in many cases, especially when training of the \u201cblack-box\u201d models and when explain of causes are more important than the outcome. In general, the PFI explains how the model characterizes the relationship between the features and the target variable. We encourage everyone try this out and apply various statistical modules and machine learning models. Also, please find the code in this github repository<\/a>.<\/p>\n
On ending note, if you\u2019re an enterprise developer who has been using C# with .NET framework and wanted to taste what machine learning development is all about, give yourself a chance with ML.NET. You can certainly reach out to Microsoft Premier service, and the team of experts will guide you step-by-step and help you to lessen the initial learning pains involved. Even without such consultation, I believe through abundance of easy-to-understand samples from Microsoft document sites and related github repos<\/a>, you will find the new joy of adventure into this new development space!<\/p>\n
 <\/p>\n","protected":false},"excerpt":{"rendered":"
ML.NET enables developers to implement machine learning tasks like classification, regression, clustering, and recommendation … etc. In this post, I will show you how to get started with ML.NET implementing permutation feature importance for employee attrition. <\/p>\n","protected":false},"author":582,"featured_media":37666,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[80,6699,2824],"tags":[4339,3],"class_list":["post-37664","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-net","category-c","category-ml","tag-ml-net","tag-team"],"acf":[],"blog_post_summary":"
ML.NET enables developers to implement machine learning tasks like classification, regression, clustering, and recommendation … etc. In this post, I will show you how to get started with ML.NET implementing permutation feature importance for employee attrition. <\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/37664","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/users\/582"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/comments?post=37664"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/37664\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media\/37666"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media?parent=37664"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/categories?post=37664"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/tags?post=37664"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}