{"id":811,"date":"2023-12-04T14:54:01","date_gmt":"2023-12-04T22:54:01","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/powerplatform\/?p=811"},"modified":"2023-12-06T11:31:21","modified_gmt":"2023-12-06T19:31:21","slug":"build-your-next-application-on-dataverse-now-with-hyperscale-support-for-any-enterprise-data","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/powerplatform\/build-your-next-application-on-dataverse-now-with-hyperscale-support-for-any-enterprise-data\/","title":{"rendered":"Build your next application on Dataverse – now with hyperscale support for any enterprise data"},"content":{"rendered":"

Dataverse, Microsoft\u2019s modern and scalable business application platform in the cloud, enables developers to build enterprise-grade applications at hyperscale. Today, Dataverse encompasses over 3.5 million Azure SQL databases and leverages Azure SQL Hyperscale<\/a> to support up to 100 TB per individual database–with rapid scale up, rapid scale out, and high availability.
\nAt Ignite, we announced Dataverse Link to Fabric<\/a>, which enables low-code developers to activate enterprise data at any scale. We are happy to announce general availability for bulk operation APIs<\/a>, improving data integration throughput and supporting even larger data estates. Dataverse supports seamless customization and extension of Dynamics 365 applications and Power Platform solutions directly by enterprise customers. Similarly, Dataverse supports integrated extensions and independent application solutions from ISVs, of which there are thousands published on Microsoft AppSource<\/a>. Additionally, Microsoft Industry Cloud<\/a> solutions are building atop Dataverse and Power Platform. To further empower developers, and customize Dynamics 365 and building Power Platform solutions, we are happy to announce general availability for:<\/p>\n

    \n
  1. Elastic Tables<\/a>, built for flexible schema and hyperscale data, while ensuring responsible AI usage, enterprise-grade security, and ease of governance.<\/li>\n
  2. Dependent assemblies<\/a> simplify development of plug-ins for implementing .NET business logic.<\/li>\n<\/ol>\n

    Enabling Hyperscale data with flexible schema using elastic tables<\/h2>\n

    Dataverse Elastic tables, powered by Azure Cosmos DB under the hood, is a powerful new option to be considered for application scenarios that entail very high scale throughput and frequent changes to the data model. Developers get all the benefits of underlying Cosmos DB, low-code application layer and granular security, without the need to maintain the infrastructure and the plumbing of application\/business tier. With its ability to massively scale horizontally using data partitioning, Elastic tables allows very high write throughput and low latency querying. Developers can take advantage of Dataverse bulk operation APIs<\/a> (CreateMultiple<\/em>, UpdateMultiple<\/em>, UpsertMultiple<\/em> and DeleteMultiple<\/em>) to bulk load data at high throughput and efficiently within the Dataverse API limits. Elastic tables are already being used extensively by Microsoft Dynamics 365 applications including storing more than 3Bn records in a single table, while enabling 120 million writes per hour and 6000 reads per second. For scenarios where the data shape changes, Elastic tables allow you to store and query data stored as JSON. You can query JSON columns<\/a> using ExecuteCosmosSqlQuery<\/a>. Additionally, when a business scenario, such as a timebound marketing campaign, requires the data to be stored only for short time periods, you can choose to automatically remove the data with time-to-live policies<\/a>. In today’s world driven by data and AI, where security and governance are non-negotiables, your data in elastic tables is protected by Dataverse security<\/a> backed by Microsoft Entra ID<\/a> and additional safeguards like customer managed key management<\/a> and lockbox<\/a>. This robust security framework for your business-critical information provides flexibility for developers to support diverse scenarios on top of already stored data, across your ecosystem. Developers can accelerate development time<\/a> by leveraging low code Power Platform<\/a>. For example, you can apply business logic using a plugin<\/a> with a Power Automate Flow<\/a>. This streamlined integration allows for the rapid creation of applications, while empowering your team to efficiently build and deploy solutions that meet your unique business needs.<\/p>\n

    \n

    “Using the Dataverse elastic table feature enabled us to keep unstructured data alongside and related to our standard Dataverse tables. This greatly reduced the dev efforts to separately integrate with any external storage and enabled faster time to market with a Power App for our application.”<\/em><\/p>\n

    David Yack, CTO and Co-founder 365.Training<\/a><\/strong><\/p>\n<\/blockquote>\n

    Read more about using Dataverse Elastic tables<\/a>.<\/p>\n

    Activate large scale enterprise data faster using Dataverse integration with Microsoft Fabric<\/h2>\n

    Dataverse\u2019s two-way integration with Microsoft Fabric<\/a>, enables low and pro code developers to generate insights from their business-critical data, without having to physically move this data. Ferranti<\/a>, who builds solutions for energy and utility companies, have reimagined their utility billing and customer engagement solution, MECOMS, with Power platform and Fabric for the future. Utilities generate a lot of data (for e.g. each meter from home or business is read every 15mins), the meter data and aggregated billing data accumulates very quickly. MECOMS was primarily used to process back-end customer bills and preserve details for many years for audit purposes. With the power of Fabric and Power Platform integration, they have expanded their product footprint to end customers served by utilities with proactive suggestions based on insights generated from the data.<\/p>\n

    \"Image<\/a><\/p>\n

    With no-copy Dataverse to Fabric integration<\/a>, Dynamics data stored in Dataverse such as customers, payments, and pricing tables, becomes available to join, merge and consolidate with meter data read from smart meters. Data is joined, processed, and stored in Microsoft OneLake using Spark and other big data tools built into Microsoft Fabric. Normalized meter data in OneLake is available for billing and managing customer interactions in Dynamics 365 without the need to copy. Developers can focus on building business logic and application without the need to orchestrate data and build pipelines saving them a lot of time and effort. MECOMS developers have built a customer-facing portal that uses detailed sensor data and billing details. Using Copilots built in Power Pages and AI plugins, consumers can ask questions and get suggestions on data generated, such as how to reduce their power consumption with ease.<\/p>\n

    \n

    \u201cWe worked closely with Microsoft at the time when Dataverse and Fabric integration was being developed. We were able to leverage the latest innovations in no-copy, no-ETL integration as well as Fabric capabilities without building features or laying out infrastructure on our own \u2013 this enabled us to focus on building and reimagining the solution that we specialize on\u2026\u201d<\/em><\/p>\n

    Ian Bruyninckx, Lead Product Architect, MECOMS, A Ferranti Company<\/strong><\/p>\n<\/blockquote>\n

    Read more about Dataverse to Fabric integration<\/a>.<\/p>\n

    New performant APIs for high throughput write operations<\/h2>\n

    We are excited to announce the general availability of new bulk operation messages<\/a> designed for very high throughput write scenarios with three new APIs CreateMultiple<\/em> and UpdateMultiple<\/em>. UpsertMultiple<\/em>. These new APIs offer a more focused approach to executing write operations as compared to ExecuteMultiple<\/em> and ExecuteTransaction<\/em> APIs, enabling batching by operation and entity for significant performance improvements. No additional cost is incurred to use these APIs and you can accomplish more write operations within the same service protection limits. Early preview customers have witnessed and reported upto 82% less time<\/strong> spent in end-to-end scenarios using CreateMultiple<\/em> and UpdateMultiple<\/em> on SQL tables. Data migration speed of over 25,000 records per second with UpsertMultiple on Elastic Tables were also observed by these customers. Customer have experienced throughput improvement of up to 5x<\/strong> using the new bulk operation messages, growing from 2 million records created per hour using ExecuteMultiple<\/em> to 10 million records in less than an hour using CreateMultiple<\/em>.<\/p>\n

    \"Image<\/a><\/p>\n

    In the snapshot above, view 2 Postman runs against the same SQL table in the same environment with no special configurations. CreateMultiple<\/em> API completes creation of 1000 records >23 times faster than Create API. Bulk operations performance on Dataverse can be tuned for a much higher throughput<\/a>.<\/p>\n

    \n

    \u201cWe, the KingswaySoft team, were very excited to hear about the Bulk API features introduced by the Microsoft Dataverse team, we saw a huge benefit of embracing the new technology. As we gained more knowledge about the new API by adding its capabilities to our Dynamics integration toolkit, we were even more impressed by the benefits that the new API brings, particularly the vast performance improvement. Overall, we think the API is well-designed and provides the benefits that every Dynamics developer should cherish and embrace, particularly when it comes to the situation where you need to work with large quantity data movement.\u201d<\/p>\n

    Daniel Cai, Managing Director of KingswaySoft<\/strong><\/p>\n<\/blockquote>\n

    The new bulk operation APIs are generally available starting today \u2014 click here<\/a> to get started!<\/p>\n

    Dataverse plug-in development & management becomes easier<\/h2>\n

    Dependent assemblies for Dataverse plug-ins<\/a> simplify plug-in development by removing the overhead of using IL Merge to manually merge dependent assemblies. With dependent assemblies, rather than registering an individual .NET assembly, you can now upload a NuGet Package that contains your plug-in assembly and any dependent assemblies. \u2022 Bring assembly dependencies of plug-in assemblies during the registration process. \u2022 Multiple plugin assemblies can be packaged into a single NuGet package. \u2022 Register the package through Power Platform Tools for Visual Studio or PAC CLI and Plug-in Registration tool (PRT) \u2022 New versions of packages can be updated in-place or registered side by side. \u2022 Assembly Full Name signing of plugin assemblies are no longer required when the assembly is part of plug-in package.<\/p>\n

    \"Image<\/a><\/p>\n

    Learn more how to use dependent assemblies for simplifying plug-in<\/a> development or watch this video<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Dataverse, Microsoft\u2019s modern and scalable business application platform in the cloud, enables developers to build enterprise-grade applications at hyperscale. Today, Dataverse encompasses over 3.5 million Azure SQL databases and leverages Azure SQL Hyperscale to support up to 100 TB per individual database–with rapid scale up, rapid scale out, and high availability. At Ignite, we announced […]<\/p>\n","protected":false},"author":137833,"featured_media":904,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-811","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-powerplatform"],"acf":[],"blog_post_summary":"

    Dataverse, Microsoft\u2019s modern and scalable business application platform in the cloud, enables developers to build enterprise-grade applications at hyperscale. Today, Dataverse encompasses over 3.5 million Azure SQL databases and leverages Azure SQL Hyperscale to support up to 100 TB per individual database–with rapid scale up, rapid scale out, and high availability. At Ignite, we announced […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/posts\/811","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/users\/137833"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/comments?post=811"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/posts\/811\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/media\/904"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/media?parent=811"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/categories?post=811"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/powerplatform\/wp-json\/wp\/v2\/tags?post=811"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}