In this blog, I will demonstrate how Text Analytics API Version 3 Preview of the Microsoft Azure Cognitive Services can be used to analyze large unstructured data. This analysis aims to understand the sentiments expressed in a solicited public comment process and determines the degree of the positivity or negativity of the comments. Text Analytics V3 expects the document in JSON with the following format: ID, text, and language. One limitation of Text Analytics V3 is that the document must have less than 5,120 characters. The data used for this demonstration is in CSV format and the comments varied from one sentence to several sentences of multiple pages. Given these constraints, I developed python codes in Jupyter notebook to do the following:<\/p>\n
- \n
- Transformed each comment line into a JSON document as expected by the API<\/li>\n
- Segmented each sentence as separate comment<\/li>\n
- Called the API to analyze the sentiment one document at a time.<\/li>\n
- Wrote the sentiment to a CSV file. This file is imported to Power BI to plot the results. A tutorial on Microsoft PowerBI<\/a> can be found here<\/a>.<\/li>\n<\/ul>\n
The Text Analytics API uses pre-trained models from Microsoft’s extensive work on NLP to analyze the comments and classify them according to the sentiments into four categories, positive, negative, mixed, and neutral. The data used for this demonstration consist of public comments for North Carolina\u2019s Medicaid Reform<\/a>. The public comment period was from June 2016 to July 2016 and a total of 1,953 responses were received. This analysis did not check for duplicate comments as it focuses on showing how Text Analytics API can be used to analyze and quantify sentiments in comments. This demonstration assumes that you have an Azure subscription and know how to create Azure Cognitive Services Resource. If you do not have an Azure subscription, you can sign up for a free trial account<\/a> and if you are unfamiliar with how to create Azure Cognitive Services, follow this tutorial<\/a>.<\/p>\n
Get the source code for this tutorial on GitHub here<\/a>.<\/p>\n
Step 1: Create Azure Resource Group<\/h2>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-32.png\")
<\/p>\nStep 2: Create Azure Cognitive Services<\/h2>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-33.png\")
<\/p>\nSupply the name, subscription, location, service plan and resource group. For the resource group, use the resource group created in Step 1 and then click Create.<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-34.png\")
<\/p>\nAfter the resource is created, click on the Go to resource<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-35.png\")
<\/p>\nStep 3: Make Note of the Key and Endpoint<\/h2>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-36.png\")
<\/p>\nNote that the Text Analytics API Version 3 Preview is only available in certain regions. As of the time of this writing (January 2020), it is available in the following regions:<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-37.png\")
<\/p>\nStep 4: Setup Python Environment in Jupyter Notebook and Load Helper Files<\/h2>\n
You can use Azure Notebook<\/a> or your preferred Jupyter Notebook like the version that is installed when you install Python from Anaconda<\/a> or Google Colab<\/a>. If you are unfamiliar with Azure Jupyter Notebook, please see this tutorial.<\/a> The Python version used for the demonstration is 3.7.<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-38.png\")
<\/p>\nStep 5: Load the Downloaded Comments File in CSV into a Data Frame<\/h2>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-39.png\")
<\/p>\nStep 6: Load the Downloaded Comments File in CSV into a Data Frame<\/h2>\n
The downloaded CSV file can be read from your desktop, OneDrive or uploaded to your Azure storage. The next statement assume you are pointing to the correct location of the file and your file is named as in the snippet. If that is not the case, you can change the statement to point to your own file.<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-40.png\")
<\/p>\nStep 7: Convert the Comments in the Data Frame to JSON Format<\/h2>\n
Some of the comments consist of multiple sentences and quite large. Passing the entire file of comments may note be optimal, so the steps below takes each line of comments and convert them to a JSON document in the formation that the Text Analytics API accepts.<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-41.png\")
<\/p>\nStep 8: Get Comment Sentiment<\/h2>\n
For each line, the code below gets the comment sentiment by passing the JSON to the Text Analytics API. The result is return for summarization<\/p>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-42.png\")
<\/p>\nStep 9: Summary of Comment Sentiments<\/h2>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-43.png\")
<\/p>\n![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-44.png\")
<\/h2>\n![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-45.png\")
<\/p>\nSentiment Summary with PowerBI<\/h2>\n
![](\"http:\/\/devblogs.microsoft.com\/premier-developer\/wp-content\/uploads\/sites\/31\/2020\/01\/word-image-46.png\")
<\/p>\n","protected":false},"excerpt":{"rendered":"In this blog, I will demonstrate how Text Analytics API Version 3 Preview of the Microsoft Azure Cognitive Services can be used to analyze large unstructured data. This analysis aims to understand the sentiments expressed in a solicited public comment process and determines the degree of the positivity or negativity of the comments.<\/p>\n","protected":false},"author":582,"featured_media":38314,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[14,25],"tags":[24,72,3],"class_list":["post-38302","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-azure","tag-azure","tag-cognitive-services","tag-team"],"acf":[],"blog_post_summary":"
In this blog, I will demonstrate how Text Analytics API Version 3 Preview of the Microsoft Azure Cognitive Services can be used to analyze large unstructured data. This analysis aims to understand the sentiments expressed in a solicited public comment process and determines the degree of the positivity or negativity of the comments.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/38302","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/users\/582"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/comments?post=38302"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/posts\/38302\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media\/38314"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/media?parent=38302"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/categories?post=38302"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/premier-developer\/wp-json\/wp\/v2\/tags?post=38302"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}