{"id":2143,"date":"2016-05-22T20:51:44","date_gmt":"2016-05-22T20:51:44","guid":{"rendered":"https:\/\/www.microsoft.com\/reallifecode\/index.php\/2016\/05\/22\/access-azure-blob-storage-from-your-apps-using-s3-api\/"},"modified":"2020-03-15T07:46:36","modified_gmt":"2020-03-15T14:46:36","slug":"access-azure-blob-storage-from-your-apps-using-s3-api","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/ise\/access-azure-blob-storage-from-your-apps-using-s3-api\/","title":{"rendered":"Access Azure Blob Storage from Your Apps using S3 Java API"},"content":{"rendered":"<p>Over the years, due to the wide adoption of Amazon S3, the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Amazon_S3#S3_API_and_competing_services\">S3 API<\/a> has become the de facto standard interface for almost all storage providers. These S3-compatible competing service providers use the standard programming interface to help customers migrate and to enable customers to write cloud-agnostic solutions. Customers use the S3 API to connect to many S3-compatible storage solutions such as Google storage, OpenStack, RiakCS, Cassandra, AliYun, and others. Currently, Azure Storage does not natively support the S3 API. This post explains in detail how we added S3 API support to Azure Storage, and how you can leverage this solution to enable your applications to store and retrieve content from various cloud storage providers.<\/p>\n<p><a href=\"https:\/\/github.com\/andrewgaul\/s3proxy\">S3Proxy<\/a> allows applications using the S3 API to access storage backends like Microsoft Azure Storage.<\/p>\n<h2 id=\"the-problem\">The Problem<\/h2>\n<p>To connect to Azure Storage, customers are required to update their existing code to use the Azure Storage SDK, which for enterprises can take awhile and S3-compatibility is one of the major features customers look for when they evaluate storage solutions side by side. This affects anyone who is currently using the S3 API and is considering Azure Storage as an alternative or hybrid solution to their existing storage backend. Companies ranging from enterprises to startups use the S3 API because it abstracts the storage backend from the apps.<\/p>\n<p><a href=\"https:\/\/github.com\/andrewgaul\/s3proxy\">S3Proxy<\/a> represents a path for customers who are blocked during onboarding and would otherwise be unwilling to rewrite basic blob management code against the Azure Storage APIs or SDKs.<\/p>\n<h2 id=\"overview-of-the-solution\">Overview of the Solution<\/h2>\n<p>We partnered with <a href=\"https:\/\/github.com\/bouncestorage\">Bounce Storage<\/a>, the maintainer of S3Proxy, and <a href=\"http:\/\/jclouds.apache.org\/\">Apache jclouds<\/a>, to enable Azure Storage support for S3Proxy. As a result of our engagement, customers can now leverage S3Proxy to reuse existing Java code with the S3 Java SDKs against Azure Blob Storage. We also provided different deployment options to allow S3Proxy to run as a containerized application anywhere, including Dokku and Cloud Foundry.<\/p>\n<p>Let\u2019s first look at how S3Proxy was implemented. Then, let\u2019s take a look at changes you need to make in your own application to store and retrieve content from Azure Storage with the S3 API.<\/p>\n<h3 id=\"how-s3proxy-works\">How S3Proxy works<\/h3>\n<p>S3Proxy leverages <a href=\"http:\/\/jclouds.apache.org\/\">Apache jclouds<\/a> BlobStore APIs to communicate with Azure Blob Storage. Like other storage providers, Azure Storage includes a <a href=\"https:\/\/msdn.microsoft.com\/en-us\/library\/dd135733.aspx\">Blob Service REST API<\/a> responsible for performing CRUD operations against containers and blobs. S3Proxy layers the S3 API on top of Azure Blob Store leveraging the abstraction that the <a href=\"http:\/\/jclouds.apache.org\/start\/blobstore\/\">jclouds BlobStore APIs<\/a> provides. S3Proxy runs as a Java web application that uses an embedded <a href=\"http:\/\/www.eclipse.org\/jetty\/\">Jetty web server<\/a>. Jetty is ideal for a solution like this as it is open source, can be embedded in the solution itself, has an extremely small footprint, and has scalable performance under heavy load.<\/p>\n<p> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2016\/05\/s3proxy.png\" alt=\"Image s3proxy\" width=\"876\" height=\"344\" class=\"aligncenter size-full wp-image-11136\" srcset=\"https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2016\/05\/s3proxy.png 876w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2016\/05\/s3proxy-300x118.png 300w, https:\/\/devblogs.microsoft.com\/ise\/wp-content\/uploads\/sites\/55\/2016\/05\/s3proxy-768x302.png 768w\" sizes=\"(max-width: 876px) 100vw, 876px\" \/><\/p>\n<h2 id=\"running-s3proxy-anywhere\">Running S3Proxy Anywhere<\/h2>\n<p>Users can use <a href=\"https:\/\/github.com\/ritazh\/s3proxydocker\">S3ProxyDocker<\/a> to test, deploy, and run S3Proxy instances as Docker containers.<\/p>\n<h3 id=\"prerequisites\">Prerequisites<\/h3>\n<ul>\n<li>Understand fundamentals of Docker and setup <a href=\"https:\/\/www.docker.com\/\">docker<\/a> locally<\/li>\n<li>Understand fundamentals of S3Proxy: configuration and setup &#8211; <a href=\"https:\/\/github.com\/andrewgaul\/s3proxy\">S3Proxy<\/a><\/li>\n<\/ul>\n<h3 id=\"getting-started\">Getting Started<\/h3>\n<ul>\n<li>The below steps assume you have Docker Toolbox installed on your local machine. If you have not done so, please follow <a href=\"https:\/\/docs.docker.com\/mac\/\">these steps<\/a>.<\/li>\n<li>Update <a href=\"https:\/\/github.com\/ritazh\/s3proxydocker\/blob\/master\/s3proxy.conf\">s3proxy.conf<\/a> with your own storage provider backend. The default s3proxy.conf is for Azure Storage.<\/li>\n<li>If you have a need for <code class=\"highlighter-rouge\">s3proxy.virtual-host<\/code>, update <a href=\"https:\/\/github.com\/ritazh\/s3proxydocker\/blob\/master\/s3proxy.conf\">s3proxy.conf<\/a> with your own docker ip.<\/li>\n<\/ul>\n<p>To find the docker ip:<\/p>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>$ docker-machine ip [docker machine name]\r\n\r\nSample output:\r\n192.168.99.100\r\n<\/code><\/pre>\n<\/div>\n<ul>\n<li>Build docker image:<code class=\"highlighter-rouge\">$ make build<\/code><\/li>\n<li>Run S3Proxy container:<code class=\"highlighter-rouge\">$ docker run -t -i -p 8080:8080 s3proxy<\/code><\/li>\n<\/ul>\n<p>If you cannot get to the internet from the container, use the following:<\/p>\n<p><code class=\"highlighter-rouge\">$ docker run --dns 8.8.8.8 -t -i -p 8080:8080 s3proxy<\/code><\/p>\n<h3 id=\"verifying-output\">Verifying Output<\/h3>\n<p>Sample output should be something like this:<\/p>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>I 12-08 01:35:30.616 main org.eclipse.jetty.util.log:186 |::] Logging initialized @1046ms\r\nI 12-08 01:35:30.642 main o.eclipse.jetty.server.Server:327 |::] jetty-9.2.z-SNAPSHOT\r\nI 12-08 01:35:30.665 main o.e.j.server.ServerConnector:266 |::] Started ServerConnector@7331196b{HTTP\/1.1}{0.0.0.0:8080}\r\nI 12-08 01:35:30.666 main o.eclipse.jetty.server.Server:379 |::] Started @1097ms\r\n<\/code><\/pre>\n<\/div>\n<p><code class=\"highlighter-rouge\">docker ps<\/code> output should be similar to this:<\/p>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>$ docker ps\r\nCONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                     NAMES\r\n789186d1debf        s3proxy                  \"\/bin\/sh -c '.\/target\"   5 seconds ago       Up 4 seconds        0.0.0.0:8080-&gt;8080\/tcp    tender_feynman\r\n<\/code><\/pre>\n<\/div>\n<p>Since we mapped port 8080 to 8080, you can navigate to [docker ip]:8080. For example: http:\/\/192.168.99.100:8080\/<\/p>\n<h3 id=\"updating-hosts-file\">Updating Hosts File<\/h3>\n<p>If you are running this locally using a local ip, you will need to update your <code class=\"highlighter-rouge\">\/etc\/hosts<\/code> file to add entries for the subdomains, which represent the containers or buckets names in your storage accounts.<\/p>\n<p>For example, if the root of the site is running at <code class=\"highlighter-rouge\">http:\/\/192.168.99.100:8080\/<\/code>, then make sure you add an entry in the <code class=\"highlighter-rouge\">\/etc\/hosts<\/code> file for each subdomain.\nIf the Azure Storage container (or S3 bucket) name is <code class=\"highlighter-rouge\">demostoragecontainer<\/code>, then add a subdomain as follows in the hosts file.<\/p>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>192.168.99.100  demostoragecontainer.192.168.99.100\r\n<\/code><\/pre>\n<\/div>\n<p>To verify, navigate to [STORAGE CONTAINER NAME].[DOCKER MACHINE IP]:8080. For example: http:\/\/demostoragecontainer.192.168.99.100:8080\/<\/p>\n<h2 id=\"enabling-your-application\">Enabling Your Application<\/h2>\n<p>To run a sample application against your S3Proxy instance, refer to <a href=\"https:\/\/github.com\/ritazh\/aws-java-sample\">the AWS Java sample app<\/a> repo to test your S3Proxy deployment. It is a simple Java application illustrating usage of the AWS S3 SDK for Java. Follow the instruction in the readme to run it.\nThis sample application connects to an S3 API compatible storage backend.<\/p>\n<ul>\n<li>It creates a container (on Microsoft Azure) or a bucket (on AWS S3).<\/li>\n<li>It lists all containers or all buckets in your storage account.<\/li>\n<li>It creates an example file to upload to a container\/bucket.<\/li>\n<li>It uploads a large file using multipart upload UploadPartRequest.<\/li>\n<li>It uploads a large file using multipart upload TransferManager.<\/li>\n<li>It lists all objects in a container\/bucket.<\/li>\n<li>It deletes an object in a container\/bucket.<\/li>\n<li>It deletes a container\/bucket.<\/li>\n<\/ul>\n<p>In summary, first make sure your S3Proxy instance is running correctly. Then run the following in your terminal to test the app:<\/p>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>mvn clean compile exec:java -Dkey=&lt;STORAGE ACCOUNT KEY&gt; -Dsecret=&lt;STORAGE ACCOUNT SECRET&gt; -Dbucketname=&lt;CONTAINER NAME TO CREATE&gt; -Ds3endpoint=&lt;S3PROXY URL&gt; -Dfilepath=&lt;FULLPATH LOCATION OF DEMO.ZIP FILE&gt;\/demo.zip\r\n<\/code><\/pre>\n<\/div>\n<p>Here is a run of the sample Java application using the S3 SDK performing basic CRUD operations against Azure blob storage using S3Proxy. On the left is a verbose log of a running S3Proxy instance. On the right, we have the sample application using S3 Java SDK doing basic CRUD operations against Azure blob storage via S3Proxy.<\/p>\n<p> <img decoding=\"async\" src=\"https:\/\/devblogs.microsoft.com\/cse\/wp-content\/uploads\/sites\/55\/2016\/05\/s3sampleapp.gif\" alt=\"Image s3sampleapp\" width=\"1256\" height=\"762\" class=\"aligncenter size-full wp-image-11137\" \/><\/p>\n<h3 id=\"updating-your-app\">Updating Your App<\/h3>\n<p>Here are a few updates you may need to add to your own app to make it work with S3Proxy.<\/p>\n<ul>\n<li>Since Azure Storage does not return the MD5 sum for ETag, make sure to include the <code class=\"highlighter-rouge\">disablePutObjectMD5Validation<\/code> flag in your app to disable MD5 check. Otherwise, it will end in error <code class=\"highlighter-rouge\">Input is expected to be encoded in multiple of 2 bytes but found: 17<\/code>.<\/li>\n<\/ul>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>System.setProperty(\"com.amazonaws.services.s3.disablePutObjectMD5Validation\", \"1\"); \r\n        \r\n<\/code><\/pre>\n<\/div>\n<ul>\n<li>Due to S3Proxy\u2019s limitations with <a href=\"https:\/\/github.com\/andrewgaul\/s3proxy\/issues\/24\">AWS signature V4<\/a>, add the following in your app to ensure v2 signature is used instead. Otherwise, you will get the <code class=\"highlighter-rouge\">Unknown header x-amz-content-sha256<\/code> exception.<\/li>\n<\/ul>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>ClientConfiguration clientConfig = new ClientConfiguration().withSignerOverride(\"S3SignerType\");\r\n<\/code><\/pre>\n<\/div>\n<ul>\n<li>To tell the AWS S3 Java SDK to use S3Proxy instead of going against an AWS region, update your <code class=\"highlighter-rouge\">AmazonS3Client<\/code> code to pass in the S3Proxy URL as the endpoint. For example, if you passed in <code class=\"highlighter-rouge\">-Ds3endpoint=:http:\/\/localhost:8000<\/code> to run this sample app, the endpoint used for <code class=\"highlighter-rouge\">AmazonS3Client<\/code> is <code class=\"highlighter-rouge\">http:\/\/localhost:8000<\/code>.<\/li>\n<\/ul>\n<div class=\"highlighter-rouge\">\n<pre class=\"highlight\"><code>AWSCredentials credentials = new BasicAWSCredentials(s3proxydemoCredentialKey, s3proxydemoCredentialSecret);\r\n        clientConfig.setProtocol(Protocol.HTTP);\r\n        AmazonS3 s3 = new AmazonS3Client(credentials, clientConfig);\r\ns3.setEndpoint(s3proxydemoEndpint);\r\n<\/code><\/pre>\n<\/div>\n<h2 id=\"other-deployment-options\">Other Deployment Options<\/h2>\n<p>You can push S3Proxy as a docker app to various platforms.<\/p>\n<h3 id=\"deploying-to-platforms-like-dokku\">Deploying to Platforms like Dokku<\/h3>\n<p><a href=\"http:\/\/dokku.viewdocs.io\/dokku\/\">Dokku<\/a> is a Docker powered open source Platform as a Service that runs on any hardware or cloud provider. Dokku can use the S3Proxy <a href=\"https:\/\/github.com\/ritazh\/s3proxydocker\/blob\/master\/Dockerfile\">Dockerfile<\/a> to instantiate containers to deploy and scale S3Proxy with few easy commands. Follow the <a href=\"https:\/\/github.com\/ritazh\/s3proxydocker\/blob\/master\/Deploy-to-Dokku.md\">Deploy-to-Dokku<\/a> guide to host your own S3Proxy in Dokku.<\/p>\n<h3 id=\"deploying-to-platforms-like-cloud-foundry\">Deploying to Platforms like Cloud Foundry<\/h3>\n<p><a href=\"https:\/\/www.cloudfoundry.org\/\">Cloud Foundry<\/a> is an open source PaaS that enables developers to deploy and scale applications in minutes, regardless of the cloud provider. Cloud Foundry with Diego can pull the S3Proxy Docker image from a Docker Registry then run and scale it as containers. Follow the <a href=\"https:\/\/github.com\/ritazh\/s3proxydocker\/blob\/master\/Deploy-to-Cloud-Foundry.md\">Deploy-to-Cloud-Foundry<\/a> guide to host your own S3Proxy in Cloud Foundry.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A proxy solution enabling customers to reuse existing code with S3 SDKs against Azure Blob Storage.<\/p>\n","protected":false},"author":21378,"featured_media":11135,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[11],"tags":[38,45,66,94,111,219,222,315,316],"class_list":["post-2143","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data","tag-amazon-s3","tag-apache-jclouds","tag-azure-blob-storage","tag-azure-storage","tag-bounce-storage","tag-java","tag-jetty","tag-s3-api","tag-s3proxy"],"acf":[],"blog_post_summary":"<p>A proxy solution enabling customers to reuse existing code with S3 SDKs against Azure Blob Storage.<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2143","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/users\/21378"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/comments?post=2143"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/posts\/2143\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media\/11135"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/media?parent=2143"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/categories?post=2143"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/ise\/wp-json\/wp\/v2\/tags?post=2143"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}