We started to transform their flagship factory as a lighthouse project to a digital platform<\/strong> to pave the way for the transformation of over 100 of their factories.<\/p>\n One dev crew from Microsoft worked on the connectivity part of the project for 15 months (organized into four project iterations). This part included collecting the OPC-UA based telemetry data on the edge within the plant, forwarding it to Azure for enrichment and storage. We also provided a rich set of APIs used by a custom user interface to view the data and manipulate the digital representation of the machines and signals in the cloud.<\/p>\n This article has the goal of outlining the project from the connectivity point of view.<\/p>\n To reduce the complexity of the Industrial Internet of Things solution architecture, we first broke it down into two parts. One part runs on the factory floor and is used to collect machine data and configure on-site resources. The second part is deployed to the cloud and makes up the data processing, enrichment, monitoring and visualization part of the system.<\/p>\n Diagram of the Industrial Internet of Things solution architecture on the Edge \/ in the factory.<\/em><\/p>\n The IoT Edge<\/strong> device is the only portion of the solution that needs to reside on the factory floor. All other components run on Microsoft Azure. Its task is to collect signals (telemetry data) from OPC-UA and non-OPC-UA devices. Non-OPC-UA devices are managed by an OPC-UA gateway which collects telemetry data from them and outputs it as OPC-UA signals. Currently, KepServerEx is in used as the gateway.<\/p>\n The OPC Publisher<\/strong> edge module, which is part of the Industrial Internet of Things platform<\/a> offered by Microsoft on GitHub, collects all data from the OPC-UA devices and gateways and sends them to IoT Hub. It is configured via a JSON file which gets pushed by the Pipeline Publisher as a cloud-to-device direct message through IoT Hub to a custom Edge module called the Configuration Controller<\/strong>. The task of the Configuration Controller is to monitor which configurations are supposed to be active, request them via the module Twin, receive them in compressed chunks – as a single cloud-to-device direct message can contain up to 256Kb of data – decompress, assemble, and write them to a shared filesystem in the JSON format expected by the OPC Publisher module. OPC Publisher detects changes to this file and updates its OPC UA node subscriptions accordingly.<\/p>\n The custom Gateway Configuration<\/strong> module is used by the Asset Registry to configure the gateway (KepServerEx) through its REST API, to onboard non-OPC-UA devices connected to the gateway. It can also connect to the gateway using opc.tcp and request sample metrics data from the newly configured devices to verify that they are working properly.<\/p>\n The OPC Twin<\/strong> edge module, also a part of the Industrial Internet of Things platform, is used to browse OPC UA devices and return their OPC UA nodes as signals they can emit to the Asset Registry. It is also controlled via cloud-to-device direct methods sent by the Asset Registry through IoT Hub.<\/p>\n The Metrics Collector<\/strong> module, a new addition to the Industrial Internet of Things Platform, collects health metrics from all Edge modules that publish data on a Prometheus endpoint and forwards them to a Log Analytics workspace. From there they are displayed in workbooks in IoT Hub, which are provided by Microsoft, in a custom dashboard that monitors the entire solution. Custom alerts are then triggered if the Edge devices are deemed unhealthy based on available disk space or system memory, device to cloud message queue length and the last time the Metrics Collector module has successfully transmitted metrics data.<\/p>\n IoT Hub Edge device metrics workbook: Edge module details and health.<\/em><\/p>\n IoT Edge device health overview in the solution\u2019s custom health dashboard.<\/em><\/p>\n The shared filesystem also hosts certificates and a chain of trust used by the OPC Twin and OPC Publisher modules to securely communicate with the OPC-UA gateway.<\/p>\n The Edge devices are automatically configured by IoT Hub with the proper Edge modules based on IoT Edge deployments in IoT Hub, as part of the release process.<\/p>\n Diagram of the Industrial Internet of Things solution architecture in the cloud.<\/em><\/p>\n The services in the cloud serve two purposes: First, the Asset Registry and Pipeline Configuration act as digital representation of all the assets in the factory that the system interacts with. Much of its functionality is exposed as a REST API which makes it a control panel of the entire system that the UI uses. Second, the Machine Data Enrichment pipeline makes the signal data from the factory, ingested as telemetry to IoT Hub or read from files by the File Signal Processor, available to the other participants in an enriched, cleaned, and understandable form.<\/p>\n The Asset Registry<\/strong> is the source of truth for the entire system. It tracks all assets and their relationships, such as plants, devices, signals, gateways, and IoT Edge devices. Any change must go through the Asset Registry, and if necessary, it will synchronize this change to the external components. For example, if the user adds a gateway device to Asset Registry, it will go ahead and configure this device on KepServerEx through the Gateway Configuration Edge module on the factory floor.<\/p>\n Internally, it\u2019s implemented as a shared Azure SQL database with several APIs:<\/p>\n The Industrial Internet of Things solution Asset Registry Data Model.<\/em><\/p>\n Machine Data Enrichment<\/strong> is a data pipeline that transforms a stream of \u201craw\u201d telemetry signals coming from the factory into the form usable by the other internal teams \/ applications.<\/p>\n We found Stream Analytics<\/strong> the ideal Azure service for this goal. It can enrich a stream of events with the reference data obtained from the Asset Registry SQL database. It then can filter or replicate the stream to multiple destinations. All signals get forwarded automatically to the hot path, an Azure Event Hub<\/strong>. Signals requiring long retention times also get sent to the cold path, an Azure Data Lake<\/strong>. Signals that couldn\u2019t be properly enriched are added to a file in Azure Storage <\/strong>for easy review. All of this happens with minimal configuration effort and with very little added latency.<\/p>\n Consumers of signal data can then choose which data source to use. For example, to assess signal quality, we save all signals from the Hot Path to Azure<\/strong> Data Explorer<\/strong> and built a Dashboard<\/strong> on top to visualize it.<\/p>\n In the middle of the engagement, we got a requirement to support a new, second source of signal data, namely CSV based. These signals are collected from devices and uploaded as files, and the Azure Data Explorer enrichment pipeline schema proved quite flexible to support this. We had two choices: add a second input to the existing Stream Analytics job or create a new Stream Analytics job to handle this new input stream. Ultimately, we chose the second option because the new event stream had a different event schema and would also arrive in large batches every few hours.<\/p>\n Another feature that the Machine Data Enrichment<\/strong> pipeline enables is tagging signals based on the use case they enable, identified by specialist that understands both the asset and the scenario. Tags are an array of predefined strings that include information about the signal in context such as: \u201cFault of type X\u201d or \u201cRelevant for end-to-end traceability\u201d. Signals can then later be queried based on their tags.<\/p>\n The solution also consists of a web-based User Interface<\/strong> which allows the customer to interact with and configure the Asset Registry and Pipeline Configuration.<\/p>\n Even though IoT Central offers a quick path for\u00a0creating powerful IoT solutions\u00a0in Azure with a PaaS offering, we have decided not to use it. The solution did not meet some of our key customer requirements: integration with AAD, device level authorization, extensibility, and geo-presence. Moreover, much of the value added by IoT Central such as visualization and reporting were already covered by a framework that the customer has in place.<\/p>\n Some gateways support forwarding events into MQTT. Leveraging MQTT would facilitate the work in publishing data to the cloud as we would be able to connect it directly to Edge Hub or Azure IoT Hub. However, the vision of the client, backed by the Azure Industrial Internet of Things architecture, is that by standardizing on OPC-UA, we would be able to eliminate the requirement of using a gateway when the shop floor machinery is updated. Moreover, OPC-UA offers features such as discoverability which enable scenarios that MQTT alone wouldn\u2019t.<\/p>\n We decided to rely on partners to translate proprietary protocols instead of relying on adding custom components from each vendor. When choosing a partner\u00a0gateway,\u00a0we decided with the customer to start with\u00a0KepServerEx as\u00a0it supports most protocols the customer has in all its worldwide factories.\u00a0The client has experience in using the product, and it exposes the configuration through a REST API which was used in the project to remotely configure the gateway.<\/p>\n The first use case for Service Bus is to consume Twin changes\u00a0from Azure IoT Hub\u00a0to\u00a0determine and coordinate data pipeline configurations being pushed to the edge.<\/p>\n Service Bus was our choice as it fulfils our requirements which include competing consumers, pub\/sub, and ordering of messages as well as alignment with the overall system architecture. We did not want to add an additional Azure service to solve a single problem.<\/p>\n Our requirements from a database point of view are to store the following information:<\/p>\n We believe that Azure Digital Twins would be a good fit to solve the family tree problem (enterprises, divisions, factories, production lines, work center, etc.). The family tree data, however, is not owned by this project, only machine data is. Therefore, the project wouldn\u2019t be using many of Azure Digital Twins\u2019 features such as feeding telemetry into twins, getting twin statuses or alerts. For this reason, we have decided not to use it.<\/p>\n The requirements we have for a database\u00a0match what a relational database offers:<\/p>\n We decided to use SQL Server as it supports the requirements and is a\u00a0product\u00a0that the customer already had experience\u00a0with.<\/p>\n A single source for enriched machine generated data as queryable database for the solution was also required. The technologies considered for this were: Azure Data Explorer, Time Series Insights, SQL Server and Cosmos DB.<\/p>\n We have chosen Azure Data Explorer because we require an append-only, low latency analytical database. ADX also offers an out of the box solution for data retention, clean-up and cost optimization with a flexible caching policy<\/a>. And, as this technology was already being used in the project, we were able to reuse the existing cluster and no additional expertise or monitoring was required.<\/p>\n Azure Data Explorer has good support for PowerBI for quick visualization of the data, but in the end, we have used Azure Data Explorer Dashboards<\/a> as it was the easiest and fastest solution for our visualization requirements.<\/p>\n The architecture allows us to run end-to-end tests entirely automated and in the cloud. The Edge device can run in a virtual machine with an added Edge module, the opc-plc server<\/a>. This is a stand-alone, easy to configure OPC-UA server that can act as the gateways or OPC-UA devices without the need of any additional infrastructure. The Edge modules can connect to it in the same way they do in the factory, just using a different address.<\/p>\n As the cloud services all expose REST APIs, the end-to-end tests can use them to provision a blank database with all the data needed, create pipeline configurations, deploy them to the Edge and monitor the connected IoT Hub\u2019s Event Hub compatible endpoint to see that the expected data is being received. With only minor changes, the same end-to-end tests can also trigger the File Based Signal processor to pick up a pre-defined file from a blob store and make sure it processes the file-based signals correctly.<\/p>\n We use several features from the Azure Monitor<\/strong> suite to build robust observability of the whole fleet:<\/p>\n We found it beneficial to add the Application Insights SDK<\/strong> to all our custom C# apps early in their development. It automatically collects distributed traces and exceptions from the application, which helps with both local and remote troubleshooting.<\/p>\n Tracing the browsing of OPC-UA devices between the Asset Registry and the Edge in Application Insights.<\/em><\/p>\n By the end of the engagement, the customer learned a lot about building industrial Internet of Things solutions using Microsoft Azure services. We from the Microsoft team learned about the challenges of running software on the Edge in a factory and interfacing with various OPC-UA servers and gateways. The project goals were all met.<\/p>\n Some lessons learned from us include that we benefited from using a domain-driven design to develop the models and build a ubiquitous language and a mutual understanding of the domain together with the customer. Also, documenting the principles with code samples has helped to educate and align the developers.<\/p>\n We chose to approach our problems in a pragmatic way and focus on simplicity while resisting the urge to over-engineer the solution and practice Conference Driven Development.<\/p>\n Consistently starting the development of new components with architectural design reviews, writing the Markdown files together using Visual Studio Live share and gathering know-how by doing short spikes to test the viability of ideas proved valuable. This ensured we included everyone from the team in the envisioning phase and allowed us to detect potential gaps or issues in our approaches that could lead to problems or dead ends later.<\/p>\n Finally, spending some initial time discussing and deciding on the project structure, naming conventions and adding linters and code styling rules early to the project helped everyone to work together very quickly. This was especially important as we were pair-programming between developers from different companies. This was further helped by constantly keeping Teams chats open that could be joined at any time by peers working on the same part of the solution.<\/p>\n Contributors to the solution from the Microsoft team listed in alphabetical order by last name: Francisco Beltrao<\/a>, Mikhail Chatillon<\/a>, Sascha Corti<\/a>, Laura Damian<\/a>, Konstantin Gukov<\/a>, Greta Jocyte<\/a>, Anaig Marechal<\/a>, Martin Weber<\/a>. We had great support from Bill Berry<\/a>, Ian Davis, and Larry Lieberman<\/a> from the CSE IoT SA Team who built a generator for a KEPServerEX configuration API library and received great feedback and support from the Microsoft IoT Team in Germany, namely Vitaly Slepakov<\/a> and Hans Gscho\u00dfmann<\/a>.<\/p>\n Azure Industrial Intrnet o Things Platform Overview:\nhttps:\/\/docs.microsoft.com\/azure\/industrial-iot\/overview-what-is-industrial-iot<\/a><\/p>\n Azure Industrial Internet of Things Platform Repository:\nhttps:\/\/github.com\/Azure\/Industrial-IoT<\/a><\/p>\n OPC Publisher Edge Module:\nhttps:\/\/azure.github.io\/Industrial-IoT\/modules\/publisher.html<\/a><\/p>\n OPC Twin Edge Module:\nhttps:\/\/azure.github.io\/Industrial-IoT\/modules\/twin.html<\/a><\/p>\nProject Goals<\/h2>\n
\n
\n
\n
\n
\n
Architecture<\/h2>\n
The Industrial Internet of Things Solution on the Edge<\/h2>\n
![\"Diagram](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2021\/11\/word-image.png\")
<\/p>\n![\"IoT](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2021\/11\/word-image-1.png\")
<\/p>\n![\"IoT](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2021\/11\/word-image-2.png\")
<\/p>\nThe Industrial Internet of Things Solution in the Cloud<\/h2>\n
![\"Diagram](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2021\/11\/graphical-user-interface-description-automaticall-8.png\")
<\/p>\n\n
![\"The](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2021\/11\/word-image-3.png\")
<\/p>\nArchitectural Design Choices<\/h2>\n
Why not use IoT Central?<\/h3>\n
Why OPC-UA and not MQTT?<\/h3>\n
Why use KepServerEx\u00a0and not\u00a0another gateway?<\/h3>\n
Service Bus vs Event Grid<\/h3>\n
SQL Server vs\u00a0Cosmos Db vs\u00a0Azure Digital Twin<\/h3>\n
\n
\n
Data Destination Storage Options \u2013 Azure Data Explorer vs Time Series Insights, SQL Server or Cosmos DB<\/h3>\n
Testability<\/h2>\n
Observability<\/h2>\n
\n
![\"Tracing](\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2021\/11\/word-image-4.png\")
<\/p>\nConclusion<\/h2>\n
Acknowledgements<\/h2>\n
Resources<\/h2>\n