App Dev Manager Khemraj Neupane and Dev Consultant Adam Toth spotlights some examples of how our team works with Premier and Unified Support customers to deliver value.

In one of our recent engagements with a large scale enterprise global customer, we were helping to resolve several issues they encountered while retrieving Office365 Management API log data for several products, including Microsoft Stream video and Power BI usage. Since these logs are stored only for seven days at the source, the customer was developing a solution that retrieved the logs for an entire day, downloaded them into a local storage on-premise, and then finally pushed them into Azure storage account for long term storage. Large blocks of PowerShell scripts were developed monolithically and scheduled to run daily to process the last 24 hours of data. The script to retrieve the log data was running abnormally slow and was taking more than 24 hours to retrieve the logs. The code was also failing to regenerate authentication tokens several times in the process.

If you are unfamiliar with the Management API, it’s essentially a mechanism to get change logs from the platform. The changes come back in paged recordsets of header information from the API, with very little details, basically just a URL that you can use to fetch further details about the change that occurred. Once you retrieve the header information for the change record, you can then call another endpoint to retrieve the details for that particular change.

Premier Developer Support Team Engagement:

As Premier Developer team started working closely with the customer and started looking into the issue, reviewing current code and reviewing current solution architecture, we found several areas for significant improvement in the architecture. First, the code itself was a big monolithic PowerShell script running and processing all data sequentially, with little to no performance metrics being captured. Once the details were retrieved, it was writing to the local disk sequentially. By adding instrumentation and timers into the script, we were able to ascertain that the slowness of retrieving the logs was not necessarily in communicating with the Office365 Management APIs (which had an average of 1.5 seconds per call to retrieve the content for each change URL, a fairly reasonable response time), but was instead the amount of time it took to persist this data to disk. With this architecture, the solution would not scale to handle enterprise level volume of logs data if they decided to expand the current workload to include other Office365 services such as Exchange, SharePoint etc. Premier Developer support suggested the customer move away from current architecture and recommended leveraging a message driven architecture in Azure in conjunction with serverless workflow and computing services for parallel processing. This would significantly boost the performance and makes the solution scalable to handle additional workloads.

The Solution:

To validate the new architecture, we will be doing a POC of the complete solution. The solution includes Azure Logic App as Jobs Manager that controls creation and execution of daily jobs, Azure Service Bus or Storage queues to store the retrieved pages of change records as messages from multiple sources like Power BI, MS Streams video, SharePoint, Exchange etc.. and Azure functions running in parallel will process each page of change records to retrieve the logs details from the source and store the retrieved data in Azure Storage account. As each page of records is retrieved, it will be queued and handled immediately by the next available thread, without having to wait first for all pages of records to be fetched. Azure key vault is used for storing client secrets for authentication requests to Management API.

Fig 1. Architectural diagram of the proposed solution

The value of Premier Developer support:

This is just one representative example of Premier Developer team helping our customers to develop highly performant and scalable solutions by doing code, design and architectural reviews. We engage with our customers in doing the assessments of current state and helping to achieve target state, developing POC’s, recommending best practices in application development, and running workshops to educate and train customers to adopt newer technologies. The Premier Developer team can help our customers in the following services in adopting cloud solutions in Azure (not an exhaustive list)-

• Infrastructure as a Service (IaaS) – VMs, Batch Service, Containers (AKS cluster, ACI)
• Storage and Database– Table, Blob, Queue and File Storage/ SQL database solution options in Azure, Azure Cosmos DB, Big Data and Data Analytics Solutions
• Application and Data Security in Azure- Authentication and Authorization solutions, data security solutions, Azure AD and AD DS in Azure
• Platform as a Service (PaaS) – App service Web App, APIs, API management, App Service Mobile app, Serverless computing (Azure Functions)
• Monitoring and Optimizing Solutions in Azure- Azure Monitor, Application Insights
• Event Based Solutions (Event grids, Event Hubs, Notification Hubs) and Message-based solutions (Azure Service Bus, Azure Storage Queue), Serverless Workflow (logic apps)
• On-Prem to Azure Migration solutions, Hybrid Solutions, Network solutions in azure, Business Continuity and DR in Azure
• Cloud native applications, containerization and microservices concepts
• SDLC Process Improvement- Azure DevOps

Interested to know more about the services provided by Microsoft Premier Developer support organization? Please reach out to us at premdevinfo@microsoft.com.