{"id":3591,"date":"2011-10-31T05:25:38","date_gmt":"2011-10-31T05:25:38","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2011\/10\/31\/tfs-databases-growing-out-of-control\/"},"modified":"2024-05-22T13:25:36","modified_gmt":"2024-05-22T20:25:36","slug":"tfs-databases-growing-out-of-control","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/tfs-databases-growing-out-of-control\/","title":{"rendered":"TFS Databases growing out of control"},"content":{"rendered":"

Several months ago I first started hearing customer reports of TFS databases growing out of control.\u00a0 Every once in a while I\u2019d hear of someone with a database growing by 100GB a week or something equally nutty.\u00a0 Previously that had been the result of someone checking in crazy amounts of data \u2013 A few years ago a similar report turned out to be someone checking in 75GB of slide decks from a conference they went to \"Image<\/a><\/p>\n

That turned out not to be the case this time.\u00a0 After some investigation, we realized that the problem was data associated the TFS\u2019s testing features.\u00a0 TFS and Microsoft Test Professional gather very rich data to enable you to do deep analysis of test failures.\u00a0 The cost is that this rich data is large.\u00a0 However, it turns out, much of this data lives long past its usefulness because we really don\u2019t have any process to clean it up.<\/p>\n

I first realized how serious it was when someone mentioned to me that on our own server, test attachment data had gotten larger than any other TFS data store.\u00a0 If you\u2019ve followed my blog over the years then you know that our server is pretty dang big (terabytes) that the version control data dominates everything else.\u00a0 When I found out that test data had passed version control data, I knew we had a problem that needed to be addressed quickly.<\/p>\n

Our first step was to release a tool we call the \u201cTest Attachment Cleaner\u201d.\u00a0 This tool enables you to define your \u201cretention policy\u201d and automate the clean up of old test attachment data.\u00a0 It helps a lot.\u00a0 Below I refer to a blog post from Anutthara that has links to all of the tools\/updates that I refer to here, including the Test Attachment Cleaner.\u00a0 Sadly one or two of our early customers with particularly larger databases and tons of test data, using the Test Attachment Cleaner discovered a bug in SQL Server that causes deleted records to not actually get deleted \u2013 and therefore space is not reclaimed.\u00a0 We worked with those customers and the SQL team to diagnose the issue, get a fix get it released.\u00a0 You\u2019ll find reference to this in Anutthara\u2019s post as well.<\/p>\n

While the Test Attachment Cleaner was a good start, it was clear that just approaching the problem by providing ways to prune out old data was not enough.\u00a0 We\u2019re clearly shoving too much data into TFS and not all of it is terribly useful or efficient.\u00a0 Hence the next step is a newly released QFE for TFS that avoids uploading binaries associated with executed tests.\u00a0 We\u2019ve found that in practice, very few people use the features that require them.\u00a0 Looking at some internal TFS databases, this saves on average about 48% of all test attachment data.\u00a0 That\u2019s half the data that will never get uploaded in the first place.<\/p>\n

We continue to investigate ways to reduce the amount of disk space used by test data \u2013 by uploading less, storing it more efficiently or pruning unused data.<\/p>\n

You can read more and get links to the various tools and updates from Anutthara\u2019s blog here: http:\/\/blogs.msdn.com\/b\/anutthara\/archive\/2011\/10\/30\/gsjgd.aspx<\/a><\/p>\n

Brian<\/p>\n","protected":false},"excerpt":{"rendered":"

Several months ago I first started hearing customer reports of TFS databases growing out of control.\u00a0 Every once in a while I\u2019d hear of someone with a database growing by 100GB a week or something equally nutty.\u00a0 Previously that had been the result of someone checking in crazy amounts of data \u2013 A few years […]<\/p>\n","protected":false},"author":244,"featured_media":14617,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[5,4],"class_list":["post-3591","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-tfs","tag-visual-studio"],"acf":[],"blog_post_summary":"

Several months ago I first started hearing customer reports of TFS databases growing out of control.\u00a0 Every once in a while I\u2019d hear of someone with a database growing by 100GB a week or something equally nutty.\u00a0 Previously that had been the result of someone checking in crazy amounts of data \u2013 A few years […]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/3591","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/comments?post=3591"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/3591\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media\/14617"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media?parent=3591"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/categories?post=3591"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/tags?post=3591"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}