August 10th, 2012

Results of NuGet CDN Experiment

A little over a week ago, we asked for your help with an experimentt that we were conducting to see whether there was value in enabling CDN support for our packages blob storage container. Over 100 of you responded from all over the world, and the data that you provided made it pretty clear that there was sufficient justification for turning on CDN support for the production NuGet blob storage container. So firstly, thank you to everyone who participated in the experiment!

In addition to turning on CDN, we wanted to share the results of the data that you provided us. First, the worldwide numbers look like the following:

  No-CDN CDN
Mean 33.8252 22.88007
Variance 601.3293 440.8645
Observations 108 108
Pearson Correlation 0.704394  
Hypothesized Mean Difference 6.8  
df 107  
t Stat 2.420113  
P(T<=t) one-tail 0.0086  
t Critical one-tail 1.659219  
P(T<=t) two-tail 0.0172  
t Critical two-tail 1.982383  

As you can see, there are some additional tests that we included so that you statisticians out there can get a deeper sense of our methodology, but at a high level, you can see that the worldwide average time to download the top 25 NuGet packages improved by over 10 seconds!

The improvements are even more pronounced depending on the specific region being considered. In the chart below, each vertical bar represents the range of values for the percentage of improvement and the horizontal line dividing each bar represents the average improvement. As you can see, the greatest improvements were seen in Australia with an average improvement of over 40%. Additionally, notable improvements can also be seen in each of the other regions from which we collected data.

CDN improvements by region

You can anecdotally draw further conclusions to explain the latency numbers by looking at the landscape of Azure CDN nodes as it exists today.

Azure CDN

All in all, the experiment was a great success, and we will be activating CDN in production very soon. We couldn’t have collected this level of data without your help, so again, many thanks!

Category
NuGet.org

0 comments

Discussion are closed.