{"id":3324,"date":"2025-02-12T09:34:48","date_gmt":"2025-02-12T17:34:48","guid":{"rendered":"https:\/\/devblogs.microsoft.com\/azure-sdk\/?p=3324"},"modified":"2025-02-12T09:34:48","modified_gmt":"2025-02-12T17:34:48","slug":"simplify-your-net-data-transfers-with-the-new-azure-storage-data-movement-library","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/azure-sdk\/simplify-your-net-data-transfers-with-the-new-azure-storage-data-movement-library\/","title":{"rendered":"Simplify your .NET data transfers with the new Azure Storage Data Movement library"},"content":{"rendered":"<p>We&#8217;re excited to announce the release of the modern Azure Storage Data Movement library, designed to simplify your data transfer experience when using Azure Blob Storage and Azure Files. This new library and the extension libraries for Azure Storage and Azure Files are now available as of February 11, 2025. In this blog post, we\u2019ll explore the benefits and how to get started with the library.<\/p>\n<h2>Why use the new and improved Data Movement library?<\/h2>\n<h3>Convenience with high performance<\/h3>\n<p>The modern Azure Storage Data Movement library is the perfect choice for uploading, downloading, and copying of blobs and files to, from, and between storage accounts. For your convenience, methods and objects are included in the client library to perform the following operations:<\/p>\n<ul>\n<li>Setting the number of parallel operations for each transfer<\/li>\n<li>Tracking transfer progress<\/li>\n<li>Pause\/resume functionality<\/li>\n<li>Checkpointing<\/li>\n<\/ul>\n<p>Compared to the standard Azure Storage client libraries, the Data Movement library targets specific advanced data transfer scenarios: directory level movement, resource consumption control, and large-scale data movement. Unlike AzCopy, PowerShell or other executable tools, the Data Movement library provides a .NET package and extension libraries that can be easily integrated into customers\u2019 on-premises or cloud service.<\/p>\n<h3>Modernization<\/h3>\n<p>The new Azure Storage Data Movement libraries share infrastructure with modern, v12 Azure Storage libraries, simplifying the process of transferring and copying data to and from Azure Storage. In addition, the libraries integrate with the modern Azure Identity libraries, helping you stay up-to-date with modern security practices.<\/p>\n<p>With the <a href=\"https:\/\/www.nuget.org\/packages\/Microsoft.Azure.Storage.DataMovement\/\">legacy version<\/a> currently on track to be retired in March 2026, the modern version of the Data Movement library supports new Azure Storage features and services going forward.<\/p>\n<h2>Get started with the new Azure Storage Data Movement library<\/h2>\n<p>The collection of modern Azure Storage Data Movement libraries includes a <a href=\"https:\/\/www.nuget.org\/packages\/Azure.Storage.DataMovement\">common client library<\/a> and specialized extension libraries for <a href=\"https:\/\/www.nuget.org\/packages\/Azure.Storage.DataMovement.Blobs\">Azure Blob Storage<\/a> and <a href=\"https:\/\/www.nuget.org\/packages\/Azure.Storage.DataMovement.Files.Shares\">Azure Files<\/a>. The common library handles core data transfer functions, while the extension libraries offer extra features specific to Azure Blob Storage and Azure Files.<\/p>\n<p>The upcoming section covers samples for getting started with the Data Movement library. Microsoft Entra token-based authentication is used in each of the samples, with the help of the Azure Identity library. For the samples, the role of Azure Storage Data Contributor (or an elevated role) is needed to perform the operations.<\/p>\n<h3>Installation<\/h3>\n<p>Use the following commands to install the required packages:<\/p>\n<pre><code class=\"language-dotnetcli\">dotnet add package Azure.Storage.DataMovement.Blobs \r\ndotnet add package Azure.Identity <\/code><\/pre>\n<h3>Add\u202f<code>using<\/code>\u202fdirectives<\/h3>\n<p>Add these\u202fdirectives to the top of your code file:<\/p>\n<pre><code class=\"language-csharp\">using Azure;\r\nusing Azure.Core;\r\nusing Azure.Identity;\r\nusing Azure.Storage.DataMovement;\r\nusing Azure.Storage.DataMovement.Blobs;<\/code><\/pre>\n<h3>The <code>TransferManager<\/code> object<\/h3>\n<p>The <code>TransferManager<\/code> serves as the primary class for initiating and managing various types of transfers, such as uploads, downloads, and copies. Let\u2019s take a look at creating a <code>TransferManager<\/code> object to interact with a local file system, Azure Blob Storage, or Azure Files:<\/p>\n<pre><code class=\"language-csharp\">TransferManager transferManager = new(new TransferManagerOptions()); <\/code><\/pre>\n<h3>Create a <code>StorageResource<\/code> object for Azure Blob Storage<\/h3>\n<p><code>StorageResource<\/code> is the foundational class for all storage resources, including blobs and files. For Azure Blob Storage, the <code>BlobsStorageResourceProvider<\/code> creates <code>StorageResource<\/code> instances for a blob container, block blob, append blob, or page blob.<\/p>\n<pre><code class=\"language-csharp\">\/\/ Create a token credential \r\nDefaultAzureCredential tokenCredential = new();\r\n\r\nBlobsStorageResourceProvider blobsProvider = new(tokenCredential); \r\n\r\n\/\/ Get a container resource \r\nStorageResource container = await blobsProvider.FromContainerAsync( \r\n    new Uri(\"http:\/\/&lt;storage-account-name&gt;.blob.core.windows.net\/sample-container\")); \r\n\r\n\/\/ Get a block blob resource - default is block blob \r\nStorageResource blockBlob = await blobsProvider.FromBlobAsync( \r\n    new Uri(\"http:\/\/&lt;storage-account-name&gt;.blob.core.windows.net\/sample-container\/sample-block-blob\"), \r\n    new BlockBlobStorageResourceOptions()); <\/code><\/pre>\n<h2>Example: Upload a local file to a blob<\/h2>\n<p>To initiate a new transfer, use the <code>TransferManager.StartTransferAsync<\/code> method. This method returns a <code>TransferOperation<\/code> object representing the transfer. The <code>TransferOperation<\/code> object allows you to monitor the transfer progress and obtain the transfer ID. The transfer ID is a unique identifier for resuming or pausing a transfer.<\/p>\n<p>Transfers involve a source and a destination, both of which are <code>StorageResource<\/code> types. The source and destination can be either <code>StorageResourceContainer<\/code> or <code>StorageResourceItem<\/code>, and they must be the same. For instance, if the source is a blob container, the destination must also be a blob container.<\/p>\n<p>The following example shows how to start a new transfer to upload a local file to a blob:<\/p>\n<pre><code class=\"language-csharp\">DefaultAzureCredential tokenCredential = new();\r\n\r\nTransferManager transferManager = new(new TransferManagerOptions()); \r\nBlobsStorageResourceProvider blobsProvider = new(tokenCredential); \r\nstring localFilePath = \"C:\/path\/to\/file.txt\"; \r\nstring blobUri = \"https:\/\/&lt;storage-account-name&gt;.blob.core.windows.net\/sample-container\/sample-blob\"; \r\n\r\nTransferOperation transferOperation = await transferManager.StartTransferAsync( \r\n    sourceResource: LocalFilesStorageResourceProvider.FromFile(localFilePath), \r\n    destinationResource: await blobsProvider.FromBlobAsync(new Uri(blobUri)));\r\n\r\nawait transferOperation.WaitForCompletionAsync();<\/code><\/pre>\n<h2>Conclusion<\/h2>\n<p>The new Azure Storage Data Movement library provides .NET developers with an easy, powerful way to facilitate data transfers in your applications using Azure Blob Storage and Azure Files.<\/p>\n<p>We encourage all users of the legacy version of the Azure Storage Data Movement library to take advantage of the significant improvements in the update and migrate to the new version.<\/p>\n<h2>Resources<\/h2>\n<ul>\n<li><a href=\"https:\/\/aka.ms\/DataMovementLibrary\">Article: Transfer data with the Data Movement library<\/a><\/li>\n<li><a href=\"https:\/\/aka.ms\/DMLCommon\">The Azure Storage Data Movement Common library on GitHub<\/a><\/li>\n<li><a href=\"https:\/\/aka.ms\/DMLBlobDoc\">Azure.Storage.DataMovement.Blobs Documentation<\/a><\/li>\n<li><a href=\"https:\/\/aka.ms\/DMLFilesDoc\">Azure.Storage.DataMovement.Files.Shares documentation<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-net\/blob\/main\/sdk\/storage\/Azure.Storage.DataMovement\/MigrationGuide.md\">Migration guide from v1 to v2 for Data Movement Library<\/a><\/li>\n<\/ul>\n<p>For feature requests, bug reports, or general support,\u202f<a href=\"https:\/\/github.com\/Azure\/azure-sdk-for-net\/issues\">open an issue<\/a>\u202fin the repository on GitHub.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This post announces the new and improved Azure Storage Data Movement library for .NET.<\/p>\n","protected":false},"author":181916,"featured_media":3325,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[701,732,738],"class_list":["post-3324","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-sdk","tag-net","tag-release","tag-storage"],"acf":[],"blog_post_summary":"<p>This post announces the new and improved Azure Storage Data Movement library for .NET.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/3324","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/users\/181916"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/comments?post=3324"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/3324\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media\/3325"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media?parent=3324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/categories?post=3324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/tags?post=3324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}