{"id":9821,"date":"2007-02-03T14:30:48","date_gmt":"2007-02-03T14:30:48","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2007\/02\/03\/managing-quality-part-1-build-quality\/"},"modified":"2018-08-14T00:34:20","modified_gmt":"2018-08-14T00:34:20","slug":"managing-quality-part-1-build-quality","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/managing-quality-part-1-build-quality\/","title":{"rendered":"Managing Quality (part 1) &#8211; Build Quality"},"content":{"rendered":"<p>Over the past several months my blog has transformed&nbsp;into more of an announcement forum.&nbsp; And while that&#8217;s a service I&#8217;m happy to provide, I&#8217;m not satisfied with it.&nbsp; I want to exchange ideas and share thoughts about topics all (ok, maybe many) software developers and teams care about.<\/p>\n<p>It&#8217;ll probably take the form of talking about the way we do things but only time will tell.&nbsp; We&#8217;ve been spending the last several months working on refining the way we measure and report on the quality of Team Foundation Server.&nbsp; We use a great many reports at many different levels of detail.&nbsp; We&#8217;re just about done with our &#8220;dashboard&#8221; reports &#8211; high level reports that give the 10,000 foot view on each of our major measuring points.&nbsp; I thought this would make a good topic to start off with.<\/p>\n<p>There are many things we measure to assess the quality.&nbsp; One of the biggest is build quality.&nbsp; It&#8217;s a high level assessment of how useful a build is and we measure it every day.&nbsp; Build health is very often an leading indicator of things going wrong on the team.&nbsp; It&#8217;s a very important metric to watch.&nbsp; Of course consistently bad builds are a sure sign of a problem.&nbsp; Consistently good builds are a good sign but not sufficient as you will see in future posts.<\/p>\n<p>We have a set of automated &#8220;scouting&#8221; steps that install a build and do a basic end-to-end run through the product: creating a team project, checking in files, adding\/modifying work items, creating a build definition and running it, viewing reports, etc.&nbsp; These tests can run in a couple of hours.&nbsp; The outcome of the scouting steps is a &#8220;build rating&#8221;.&nbsp; These rating are:<\/p>\n<ul>\n<li><strong>Self Test<\/strong> &#8211; The build is good and ready for in depth testing.  \n<li><strong>Self Test with Work arounds<\/strong> &#8211; The build is good but requires manual work arounds.&nbsp; With these work arounds all functional areas pass scouting tests.  \n<li><strong>Partially Testable<\/strong> &#8211; Some functional areas may fail and further testing may be blocked for that area but most work and none of the failures block overall progress.  \n<li><strong>Self Toast<\/strong> &#8211; The build is no good.&nbsp; It won&#8217;t install, can&#8217;t create Team Projects, or other major features are not working properly.  \n<li><strong>Build failure<\/strong> &#8211; There was a break and a build could not be produced.<\/li>\n<\/ul>\n<p>Every day we rate the build based on the assessment with a&nbsp;report that looks like this (this was a build that was not very good):<\/p>\n<p>Build 20201.00 \u2013 Partially Testable<\/p>\n<p>1) Team Build SKU fails to install (Product Bug 188908) causing the Setup and Reporting features to be partially testable and Team Build to be self-toast<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image%7B0%7D%5B3%5D.png\"><img decoding=\"async\" style=\"border-right: 0px;border-top: 0px;border-left: 0px;border-bottom: 0px\" height=\"240\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/image%7B0%7D_thumb%5B1%5D.png\" width=\"249\" border=\"0\"><\/a> <\/p>\n<p>&nbsp;<\/p>\n<p>Of course it&#8217;s easy to get lost in that much daily detail so we have created some dashboard reports to capture Build Quality trend information.&nbsp; Here&#8217;s a recent report:<\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image001%5B8%5D.jpg\"><img decoding=\"async\" style=\"border-right: 0px;border-top: 0px;border-left: 0px;border-bottom: 0px\" height=\"84\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image001_thumb%5B3%5D.jpg\" width=\"720\" border=\"0\"><\/a><\/p>\n<p><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image001%5B1%5D%5B2%5D.jpg\"><img decoding=\"async\" style=\"border-right: 0px;border-top: 0px;border-left: 0px;border-bottom: 0px\" height=\"208\" alt=\"cid:image002.jpg@01C74013.C5DA1AB0\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image001%5B1%5D%5B1%5D.jpg\" width=\"240\" border=\"0\"><\/a><\/p>\n<p>Each row in the chart is a different branch of the code that we maintain.&nbsp; We are currently building\/testing 3 branches:<\/p>\n<ul>\n<li><strong>Orcas<\/strong> &#8211; The active development (what we call a product unit branch or PU Branch for short) for the Orcas release.  \n<li><strong>Rosario<\/strong>&nbsp;&#8211; Yes, I know I&#8217;m not supposed to use that name but it&#8217;s in the chart and I didn&#8217;t want to cut it out.&nbsp; This is the active development for the Rosario release.&nbsp; One of the reasons this is so much better quality than the Orcas branch right now is that we have a lot fewer people working on it.  \n<li><strong>Main<\/strong> &#8211; This is the branch from which all official Visual Studio builds are produced.&nbsp; Each product unit (or in some cases groups of product units) have branches off of main (the Orcas branch above is a PU Branch for the Team System team) and periodically &#8220;reverse integrate&#8221; their work into Main when it is ready to be shared with the entire division.<\/li>\n<\/ul>\n<p>From this report, you can see that since early December the Orcas branch has been rocky.&nbsp; In early January we got very concerned about this and started jumping up and down about fixing the build quality.&nbsp; This report is a great way to see that things are headed for the ditch.<\/p>\n<p>Sometimes you want to drill in even further and see more detail for build issues.&nbsp; Here&#8217;s a report with one more level of drill down.&nbsp; Here you can see which components are having issues and which are not.<\/p>\n<p><b><a href=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image001%5B2%5D%5B4%5D.jpg\"><img decoding=\"async\" style=\"border-right: 0px;border-top: 0px;border-left: 0px;border-bottom: 0px\" height=\"234\" src=\"https:\/\/devblogs.microsoft.com\/wp-content\/uploads\/sites\/8\/2019\/02\/clip_image001%5B2%5D_thumb%5B1%5D.jpg\" width=\"720\" border=\"0\"><\/a><\/b><\/p>\n<p>&nbsp;<\/p>\n<p>Build quality is our first line of defense.&nbsp; It&#8217;s the first thing we measure and the first place we look for an assessment of overall quality health.&nbsp; If it&#8217;s bad none of the other stuff I&#8217;m going to show you in the coming days will matter.<\/p>\n<p>Well, that&#8217;s an overview of our build quality assessment.&nbsp; I can see now this is going to be a really long series of posts.&nbsp; This has only scratched the surface of how we manage quality on the TFS team.&nbsp; I&#8217;ll try to post them as rapidly as I can and maybe follow up with a summary that shows one full daily dashboard report.<\/p>\n<p>I know some of you are likely to a\nsk me for these reports.&nbsp; I&#8217;ll see what I can do but some of these are tied into our methodology enough that I don&#8217;t know that I can get them easily separated out.&nbsp; I&#8217;m not going to even try until I&#8217;m done with the series.&nbsp; When it&#8217;s all done we can have a talk about what you find most compelling and whether on not there&#8217;s anything we can share for your own use.<\/p>\n<p>You might ask, &#8220;Hey, don&#8217;t you use the reports in the process templates you ship?&#8221;&nbsp; The answer is yes, some of them.&nbsp; But our process (just like yours) is evolving all of the time.&nbsp; Many of these reports have been created in the past six months.&nbsp; We&#8217;ll take the ones that work best and incorporate them into the product in future releases.<\/p>\n<p>&nbsp;<\/p>\n<p>Thanks,<\/p>\n<p>Brian<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Over the past several months my blog has transformed&nbsp;into more of an announcement forum.&nbsp; And while that&#8217;s a service I&#8217;m happy to provide, I&#8217;m not satisfied with it.&nbsp; I want to exchange ideas and share thoughts about topics all (ok, maybe many) software developers and teams care about. It&#8217;ll probably take the form of talking [&hellip;]<\/p>\n","protected":false},"author":244,"featured_media":14617,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1],"tags":[5],"class_list":["post-9821","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-tfs"],"acf":[],"blog_post_summary":"<p>Over the past several months my blog has transformed&nbsp;into more of an announcement forum.&nbsp; And while that&#8217;s a service I&#8217;m happy to provide, I&#8217;m not satisfied with it.&nbsp; I want to exchange ideas and share thoughts about topics all (ok, maybe many) software developers and teams care about. It&#8217;ll probably take the form of talking [&hellip;]<\/p>\n","_links":{"self":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/9821","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/users\/244"}],"replies":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/comments?post=9821"}],"version-history":[{"count":0,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/posts\/9821\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media\/14617"}],"wp:attachment":[{"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/media?parent=9821"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/categories?post=9821"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/devblogs.microsoft.com\/bharry\/wp-json\/wp\/v2\/tags?post=9821"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}