Several months ago I first started hearing customer reports of TFS databases growing out of control.  Every once in a while I’d hear of someone with a database growing by 100GB a week or something equally nutty.  Previously that had been the result of someone checking in crazy amounts of data – A few years ago a similar report turned out to be someone checking in 75GB of slide decks from a conference they went to

That turned out not to be the case this time.  After some investigation, we realized that the problem was data associated the TFS’s testing features.  TFS and Microsoft Test Professional gather very rich data to enable you to do deep analysis of test failures.  The cost is that this rich data is large.  However, it turns out, much of this data lives long past its usefulness because we really don’t have any process to clean it up.

I first realized how serious it was when someone mentioned to me that on our own server, test attachment data had gotten larger than any other TFS data store.  If you’ve followed my blog over the years then you know that our server is pretty dang big (terabytes) that the version control data dominates everything else.  When I found out that test data had passed version control data, I knew we had a problem that needed to be addressed quickly.

Our first step was to release a tool we call the “Test Attachment Cleaner”.  This tool enables you to define your “retention policy” and automate the clean up of old test attachment data.  It helps a lot.  Below I refer to a blog post from Anutthara that has links to all of the tools/updates that I refer to here, including the Test Attachment Cleaner.  Sadly one or two of our early customers with particularly larger databases and tons of test data, using the Test Attachment Cleaner discovered a bug in SQL Server that causes deleted records to not actually get deleted – and therefore space is not reclaimed.  We worked with those customers and the SQL team to diagnose the issue, get a fix get it released.  You’ll find reference to this in Anutthara’s post as well.

While the Test Attachment Cleaner was a good start, it was clear that just approaching the problem by providing ways to prune out old data was not enough.  We’re clearly shoving too much data into TFS and not all of it is terribly useful or efficient.  Hence the next step is a newly released QFE for TFS that avoids uploading binaries associated with executed tests.  We’ve found that in practice, very few people use the features that require them.  Looking at some internal TFS databases, this saves on average about 48% of all test attachment data.  That’s half the data that will never get uploaded in the first place.

We continue to investigate ways to reduce the amount of disk space used by test data – by uploading less, storing it more efficiently or pruning unused data.

You can read more and get links to the various tools and updates from Anutthara’s blog here: http://blogs.msdn.com/b/anutthara/archive/2011/10/30/gsjgd.aspx

Brian