When transferring data with the Azure Storage client libraries, a lot is happening behind-the-scenes. These workings can affect speed, memory usage, and sometimes whether the transfer succeeds. This post will help you get the most out of Storage client library data transfers.<\/p>\n
These concepts apply to the Azure.Storage.Blobs<\/a> and Azure.Storage.Files.DataLake<\/a> packages. Specifically, we’re looking at APIs that accept StorageTransferOptions<\/a> as a parameter. Commonly used examples are:<\/p>\n You can also define a value for While the class contains nullable values, the client libraries will use defaults for each individual value when not provided. These defaults are fine in a data center environment, but likely unsuitable for home consumer environments. Poorly tuned The Storage client libraries will split a given upload stream into various subuploads based on provided Note: block blobs have a maximum block count of 50,000. Your blob, then, has a maximum size of 50,000 times The Storage REST layer doesn’t support picking up a REST upload where you left off. Individual transfers are either completed or lost. To ensure resiliency, if a stream isn’t seekable, the Storage client libraries will buffer the data for each individual REST call before starting the upload. Outside of network speed, this behavior is also why you may be interested in setting a smaller value for If uploading with parallel REST calls to maximize network throughput, the client libraries need sources they can read from in parallel. Since streams are sequential, when uploading in parallel, the Storage client libraries will buffer the data for each individual REST call before starting the upload even if the provided stream is already seekable<\/strong>.<\/p>\n To avoid the Storage client libraries buffering your data for upload, you must provide a seekable stream and ensure When a seekable stream is provided, its length is checked against this value. If the stream length is within this value, the entire stream will be uploaded as a single REST call. Otherwise, upload will be done in parts as described previously in this document.<\/p>\n Note: when using The Storage client libraries will split a given download request into various subdownloads based on provided Receiving multiple HTTP responses simultaneously with body contents will have memory implications. However, the Storage client libraries don’t explicitly add a buffer step for downloaded contents. Incoming responses are processed in order. The client libraries configure a 16-kilobyte buffer for copying streams from HTTP response stream to caller-provided destination stream\/file path.<\/p>\n The Storage client libraries will make one download range request using <\/p>\n Thanks for reading this Azure SDK blog post. We hope you learned something new, and we welcome you to share the post. We’re open to Azure SDK blog contributions from our readers. To get started, contact us at azsdkblog@microsoft.com<\/a> with your idea, and we’ll set you up as a guest blogger.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" Learn how to get better performance out of your Azure Storage transfers and avoid timeouts.<\/p>\n","protected":false},"author":42728,"featured_media":1722,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[701,750,706,703,738],"class_list":["post-1719","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-azure-sdk","tag-net","tag-azure-sdk","tag-azuresdk","tag-clientlibraries","tag-storage"],"acf":[],"blog_post_summary":" Learn how to get better performance out of your Azure Storage transfers and avoid timeouts.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/1719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/users\/42728"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/comments?post=1719"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/posts\/1719\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media\/1722"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/media?parent=1719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/categories?post=1719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/azure-sdk\/wp-json\/wp\/v2\/tags?post=1719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
BlobClient.UploadAsync(Stream stream, ...)<\/code><\/li>\nBlobClient.UploadAsync(string path, ...)<\/code><\/li>\nBlobClient.DownloadToAsync(Stream stream, ...)<\/code><\/li>\nBlobClient.DownloadToAsync(string path, ...)<\/code><\/li>\nDataLakeFileClient.UploadAsync(Stream stream, ...)<\/code><\/li>\nDataLakeFileClient.UploadAsync(string path, ...)<\/code><\/li>\nDataLakeFileClient.ReadToAsync(Stream stream, ...)<\/code><\/li>\nDataLakeFileClient.ReadToAsync(string path, ...)<\/code><\/li>\n<\/ul>\nStorageTransferOptions<\/code><\/h2>\nStorageTransferOptions<\/code> is the key class for tuning your performance. Storage transfers are partitioned into several subtransfers based on the values in this class. Here, you define values for the following properties, which are the basis for managing your transfer:<\/p>\n\n
MaximumConcurrency<\/code>: the maximum number of parallel subtransfers that can take place at once.\n\n
Azure.Storage.Blobs<\/code> 12.10.0 and Azure.Storage.Files.DataLake<\/code> 12.8.0), only asynchronous operations can parallelize transfers. Synchronous operations will ignore this value and work in sequence.<\/li>\nMaximumTransferSize<\/code>: the maximum data size of a subtransfer, in bytes.\n\n
InitialTransferSize<\/code>. Unlike the name suggests, your MaximumTransferSize<\/code> does not<\/strong> limit this value. In fact, you often want InitialTransferSize<\/code> to be at least<\/em> as large as your MaximumTransferSize<\/code>, if not larger. InitialTransferSize<\/code> defines a separate data size limitation for an initial attempt to do the entire operation at once with no subtransfers. Using a single transfer cuts down on overhead, leading to faster transfers for some data lengths based on your MaximumTransferSize<\/code>. If unsure of what’s best for you, setting this property to the same value used for MaximumTransferSize<\/code> is a safe option.<\/p>\nStorageTransferOptions<\/code> can result in excessively long operations and even timeouts. You should always be proactive in determining your values for this class.<\/p>\nUploads<\/h2>\n
StorageTransferOptions<\/code>, each with their own dedicated REST call. With BlobClient<\/code>, this operation will be Put Block<\/a> and with DataLakeFileClient<\/code>, this operation will be Append Data<\/a>. The Storage client libraries manage these REST operations in parallel (depending on transfer options) to complete the total upload.<\/p>\nMaximumTransferSize<\/code>.<\/em><\/p>\nBuffering on uploads<\/h3>\n
MaximumTransferSize<\/code> even when uploading in sequence. MaximumTransferSize<\/code> is the maximum division of data to be retried after a connection failure.<\/p>\nMaximumConcurrency<\/code> is set to 1. While this strategy should suffice in most situations, your code could be using other features of the client libraries that require buffering anyway. In this case, buffering will still occur.<\/p>\nInitialTransferSize<\/code> on upload<\/h3>\nBlobClient<\/code>, an upload within the InitialTransferSize<\/code> will be performed using Put Blob<\/a>, rather than Put Block.<\/em><\/p>\nInitialTransferSize<\/code> has no effect on an unseekable stream and will be ignored.<\/p>\nDownloads<\/h2>\n
StorageTransferOptions<\/code>, each with their own dedicated REST call. The client libraries manage these REST operations in parallel (depending on transfer options) to complete the total download.<\/p>\nBuffering on downloads<\/h3>\n
InitialTransferSize<\/code> on download<\/h3>\nInitialTransferSize<\/code> before anything else. Upon downloading that range, total resource size will be known. If the initial request downloaded the whole content, we’re done! Otherwise, the download steps described previously will begin.<\/p>\nSummary<\/h2>\n
StorageTransferOptions<\/code> contains the tools to optimize your transfers. It provides options that affect transfer speeds and memory usage. Unless you’re working with trivial file sizes, be proactive in configuring these options based on the environment in which your client will run.<\/p>\nAzure SDK Blog Contributions<\/h2>\n
\n