Plugins in Dataverse

Sweta Kumari

Working with Dataverse

Introduction

In a recent engagement, our customers expressed the need to develop an application for synchronizing data between their Engagement Banking Platform (EBP) and the Unified Customer Profile (UCP). The underlying technology for UCP is Dataverse. Dataverse provides a number of technology options for exporting data based on an event driven architecture. After evaluating several options, we opted for webhooks because of its simplicity, real-time data synchronization, and ease of implementation. Hence, we delved into the process of programmatically registering webhook plugins using the Dataverse Web API.

Recognizing the value of our findings, we decided to consolidate our experiences into a comprehensive blog post, with the aim of assisting others facing similar hurdles. This blog post covers various data export options from Dataverse, providing a comparative analysis based on different parameters. Additionally, it offers detailed guidance on the procedure for programmatically registering plugins using the Dataverse Web API.

Data Export Options

There are a number of proven 1st party (Microsoft) technology options for exporting data out of Dataverse based on an event-driven architecture.

• Logic Apps
• Power Automate
• Custom Azure-aware Plugins
• Webhook Plugins
• Service Bus Plugins

Logic Apps

Azure Logic Apps provide a low-code/no-code approach to automating workflows. Dataverse connectors within Logic Apps enable seamless data integration and export. Logic Apps are well-suited for scenarios where complex workflows are needed, and their visual design interface makes them accessible to users with varying technical expertise.

Power Automate

Power Automate, formerly known as Microsoft Flow, is a cloud-based service that allows users to automate workflows across various applications and services. With Dataverse connectors, Power Automate enables users to design automated data export processes without the need for extensive coding. It is a user-friendly option for smaller-scale data exports and routine tasks.

Custom Azure-aware Plugins

Custom Azure-aware Plugins offer a high degree of customization by integrating directly with Azure services. Leveraging Azure Functions or Azure Logic Apps, these plugins can be tailored to specific export requirements. While this approach provides flexibility, it involves more development effort and maintenance compared to other options.

Webhook Plugin

Webhook Plugins offer a straightforward way to trigger actions in external systems when specific events occur in Dataverse. By leveraging HTTP callbacks, Webhook Plugins enable the seamless transfer of data to external endpoints. While this method is efficient for real-time data synchronization, scaling might require additional infrastructure and load balancing.

Service Bus Plugins

Service Bus Plugins provide a reliable messaging infrastructure for communication between Dataverse and external systems. Using Azure Service Bus, data export operations can be decoupled from the core Dataverse processes, ensuring scalability and reliability. This method is particularly suitable for handling large volumes of data and achieving asynchronous data export.

Comparison of Data-Export Options

Comparing the above options with respect to different parameters:

Scalability

• Service Bus Plugins and Custom Azure-aware Plugins are highly scalable, making them suitable for handling large volumes of data.
• Webhook Plugins, Logic Apps, and Power Automate are more suitable for smaller scale exports.

Complexity and Customization:

• Custom Azure-aware Plugins offer the highest level of customization but require more development effort.
• Logic Apps and Power Automate offer a balance between customization and ease of use.
• Webhook Plugins are simpler but may lack some advanced customization options.

Real-time vs. Batch Processing:

• Webhook Plugins are well-suited for real-time data synchronization.
• Service Bus Plugins and Custom Azure-aware Plugins are suitable for both real-time and batch processing.
• Logic Apps and Power Automate are better suited for real-time or near-real-time scenarios.

User Accessibility:

• Power Automate and Logic Apps are designed for users with varying technical expertise, providing a more accessible option for non-developers.
• Custom Azure-aware Plugins and Service Bus Plugins may require more technical expertise for configuration and maintenance.
• Webhook Plugins are relatively simple but may still require development knowledge.

For our use-case, we chose Webhooks Plugin as the Data Exporter for several key reasons:

• Real-time integration, it offers real-time event notifications which helps with use cases that require real-time processing.
• Minimum tech footprint, apart from a new registration process in Dataverse, no additional technology is required.
• No custom development, the Dataverse outbound event registration is a one off setup, programmatically or by configuration, there is no additional custom development required from the Dataverse side of the solution.
• Lower total cost of ownership for customers, the only component of the end-to-end data sync service solution that resides in the customer tenant is the actual webhook Plugin registration deployed to the target Dataverse instance, this deployment can also be automated.

How to register plugins programmatically using Dataverse Web API

Prerequisites

• Before you begin, ensure you have access to a Dataverse environment and the necessary permissions to configure webhooks.
• To interact with Dataverse through the REST API, you need to authenticate your requests. Ensure that you have the necessary authentication credentials – clientId, clientSecret, tenantId and resource to interact with the Dataverse API.

Steps to register Webhook Plugin

• To register WEBHOOK you need the following:

  1. Set the variables that will be used by the commands:

     API_BASE_URL = 'https://<env-name>.api.<region>.dynamics.com/api/data/v9.2'
     ACCESS_TOKEN = <TOKEN FOR CONNECTING TO DATAVERSE>
     WEBHOOK_DATA = '{
          "name": "<Webhook Plugin Name>",
          "url": "<Endpoint url which will receive webhook events>",
          "contract": 8,
          "authtype": 5,
          "authvalue": "<settings><setting name=\"<PLACEHOLDER FOR KEY>\" value=\"<PLACEHOLDER FOR VALUE>\" /></settings>"
      }'

     Refer here for allowed values for contract and authtype.

  2. Make the below POST API call:

     RESPONSE=`curl -X POST "$API_BASE_URL/serviceendpoints" \
     -H "Content-Type: application/json" \
     -H "Accept: application/json" \
     -H "Authorization: Bearer $ACCESS_TOKEN" \
     -H "OData-MaxVersion: 4.0" \
     -H "OData-Version: 4.0" \
     -H "Prefer: return=representation" \
     --data "$WEBHOOK_DATA"`

  3. Retrieve the ServiceEndpointId from the RESPONSE, needed for registering STEP.

     ENDPOINT_ID=$(echo "$RESPONSE" | jq -r '.serviceendpointid')

  4. Next, register STEP within the above registered Webhook. Follow this

Steps to register ServiceBus Plugin

• To register SERVICE BUS you need the following:

  1. Set the variables that will be used by the commands:

     API_BASE_URL = 'https://<env-name>.api.<region>.dynamics.com/api/data/v9.2'
     ACCESS_TOKEN = <TOKEN FOR CONNECTING TO DATAVERSE>
     SERVICE_BUS_DATA = '{
          "name": "<Service Bus Plugin Name>",
          "saskeyname": "<SAS Key Name>",
          "contract": 6,
          "messageformat": 2,
          "namespaceaddress": "<Full Service Bus Namespace>",
          "authtype": 2,
          "path": "<QUEUE NAME>",
          "saskey": "<Primary SAS Key>"
      }'

     Refer here for allowed values for contract and authtype.

  2. Make the below POST API call:

     RESPONSE=`curl -X POST "$API_BASE_URL/serviceendpoints" \
     -H "Content-Type: application/json" \
     -H "Accept: application/json" \
     -H "Authorization: Bearer $ACCESS_TOKEN" \
     -H "OData-MaxVersion: 4.0" \
     -H "OData-Version: 4.0" \
     -H "Prefer: return=representation" \
     --data "$SERVICE_BUS_DATA"`

  3. Retrieve the ServiceEndpointId from the RESPONSE, needed for registering STEP.

     ENDPOINT_ID=$(echo "$RESPONSE" | jq -r '.serviceendpointid')

  4. Next, register STEP within the above registered Service bus plugin. Follow this

Register Step within the registered Plugin

• To register STEP within the plugins you need the following:

  1. Set the variables that will be used by the commands:

     EVENT_HANDLER = "/serviceendpoints($ENDPOINT_ID)"
     STEP_DATA = '{
        "name": "<Step Name>",
        "description": "<Description for the Step>",
        "eventhandler_serviceendpoint@odata.bind":"'$EVENT_HANDLER'",
        "sdkmessageid@odata.bind":"/sdkmessages(<GUID of crud operation>)",
        "sdkmessagefilterid@odata.bind": "/sdkmessagefilters(<Guid of Table>)",
        "supporteddeployment":0,
        "asyncautodelete": false,
        "filteringattributes": "<comma-separated column names>",
        "rank": 1,
        "mode": 1,
        "stage": 40
      }'

  2. Make the below POST API call:

     RESPONSE=`curl -X POST "$API_BASE_URL/sdkmessageprocessingsteps" \
     -H "Content-Type: application/json" \
     -H "Accept: application/json" \
     -H "Authorization: Bearer $ACCESS_TOKEN" \
     -H "OData-MaxVersion: 4.0" \
     -H "OData-Version: 4.0" \
     -H "Prefer: return=representation" \
     --data "$STEP_DATA"`

How to retrieve respective GUID values while registering STEP (Windows-only)

• To retrieve the corresponding GUID value for sdkmessageid and sdkmessagefilterid follow these steps:

  1. Register Webhook and the step using PRT(Plugin Registration Tool) as here
  2. Retrieve ENDPOINT_ID by selecting the Webhook in the PRT and copying the value of ServiceEndpointId from the Properties tab as below.

  3. Next, make a GET api call to get the details of the STEP registered using PRT.

     • Set the variables that will be used by the commands:

       STEP_PARAMS="/serviceendpoints("$ENDPOINT_ID")/serviceendpoint_sdkmessageprocessingstep"

     • Make the below GET api call:

       RESPONSE=`curl "$API_BASE_URL$STEP_PARAMS" \
      -H "Authorization: Bearer $ACCESS_TOKEN" \
      -H "Accept: application/json" \
      -H "OData-MaxVersion: 4.0" \
      -H "OData-Version: 4.0"`

     • Retrieve the value of _sdkmessageid_value and _sdkmessagefilterid_value from the above RESPONSE.

Conclusion

Choosing the right method for exporting data from Dataverse depends on the specific requirements. For smaller-scale exports or routine tasks, Power Automate or Logic Apps may be sufficient. For larger-scale and more complex scenarios, Service Bus Plugins or Custom Azure-aware Plugins offer greater scalability and customization. If simplicity, real-time data synchronization, and ease of implementation are your top priorities, Webhook Plugins emerge as a preferred option. Consider the trade-offs between simplicity, scalability, and customization when selecting the most appropriate method for your data export needs in Dataverse.

Sweta Kumari Software Developer

Follow