{"id":3021,"date":"2012-08-28T08:13:45","date_gmt":"2012-08-28T08:13:45","guid":{"rendered":"https:\/\/blogs.msdn.microsoft.com\/bharry\/2012\/08\/28\/tfs-shipping-cadence\/"},"modified":"2024-05-01T12:52:44","modified_gmt":"2024-05-01T19:52:44","slug":"tfs-shipping-cadence","status":"publish","type":"post","link":"https:\/\/devblogs.microsoft.com\/bharry\/tfs-shipping-cadence\/","title":{"rendered":"TFS Shipping Cadence"},"content":{"rendered":"

I\u2019ve been meaning to write about this for a while but somehow the days just slip by and I never find the time.<\/p>\n

Team Foundation Service<\/h3>\n

If you are a reader of my blog then you\u2019ve been seeing my posts on our service updates for months now.\u00a0 But let me rewind a bit and start at the beginning.<\/p>\n

About 2 years ago we began the journey to bring TFS to the cloud.\u00a0 In the beginning it was just an experiment \u2013 Can we port TFS to Azure and have a viable cloud hosted solution?\u00a0 It took a summer to prove that we could do it and the fall\/winter to shore it up and make it production ready.\u00a0 So a little over a year ago, we decided we were serious about this and started asking what a product plan for it looked like.<\/p>\n

Obviously our background is as an on-premises mission critical server team \u2013 we\u2019ve been doing that for 10 years.\u00a0 Further, we\u2019re part of the Microsoft \u201cmachine\u201d and that has it\u2019s own set of ingrained practices.\u00a0 We shipped on 2-3 year cycles.\u00a0 I believe we had\/have a very good 2-3 year cycle with strong customer engagement, good planning, agile release mechanisms (like Power Tools), etc but still \u2013 it\u2019s a 2-3 year cycle.\u00a0 We knew going into the cloud space, that wasn\u2019t going to work.<\/p>\n

In the cloud, people expect freshness.\u00a0 They expect progress.\u00a0 If your site\/app hasn\u2019t progressed recently, people start to assume it\u2019s dead.\u00a0 We needed to rethink a lot about what we do.\u00a0 That thinking started with trying to figure out what we wanted.\u00a0 Like people often do, we started with what we knew and tried to evolve from there.\u00a0 We spent a few months thinking through \u201cWhat if we do major releases every year and minor releases every 6 months?\u201d,\u00a0 \u201cMajor releases every 6 months, patches once a month?\u201d, \u201cWhat if we do quarterly releases \u2013 can we get the release cycle going that fast?\u201d, etc.\u00a0 We spent time debating what constitutes a major release vs a minor release.\u00a0 How much churn are customers willing to tolerate?\u00a0 We went round and round.\u00a0 Ultimately we concluded we were just approaching the question wrong.\u00a0 When a change this big is necessary \u2013 forget were you are and just ask where you want to be and then ask what it would take to get there.<\/p>\n

So late last summer, we were shipping service updates about every 4-6 months and we made the call that we were going to go to weekly updates.\u00a0 The goal was to ship new features (not just bug fixes) every week.\u00a0 To some degree it was a statement.\u00a0 Let\u2019s figure out how fast we can go.\u00a0 Some asked why not every day? \u2013 certainly some services out there do that.\u00a0 Ultimately I felt that it just wasn\u2019t necessary for our product\/customer base\/size of team.\u00a0 Maybe someday that will make sense but I didn\u2019t feel it did for where we are today.\u00a0 To avoid having the capabilities delivered in this weekly cadence be random, we decided on a 6 month planning horizon.\u00a0 We\u2019d plan in roughly 6 month chunks and then deliver in weekly increments.<\/p>\n

We started working through this last fall (I\u2019ll write more about this effort later) and gradually turned up the frequency from 4-6 month updates.\u00a0 The team was already executing with a Scrum based process, using aligned 3 week sprints.\u00a0 As our release cycles got shorter and shorter, we realized that those 3 week sprints actually formed a natural cadence for the team.\u00a0 We plan the sprint, we build it, we deliver a \u201cpotentially shippable increment\u201d \u2013 except, in this case, it wasn\u2019t \u201cpotentially\u201d; now it really was \u201cshippable\u201d.\u00a0 Because of this natural alignment, we decided to stop the cycle tuning process at 3 weeks and ship feature updates to the service every 3 weeks rather than every week.\u00a0 We knew we still needed ways to update the service to resolve high priority issues more frequently than that, so we instituted a \u201cPatch Monday\u201d plan that says any given Monday we can, if needed, roll out important but not urgent service fixes.\u00a0 We also can roll out urgent fixes any given day, and sometimes do \u2013 literally within a few hours of discovering the issue.<\/p>\n

The last piece that came in to place was realizing that 6 month planning wasn\u2019t even really enough to make sure the team really had a clear view of where we were going so we added, a 12-18 month \u201cvision\u201d to make sure we were all rowing in the same direction.<\/p>\n

So our cadence is:<\/p>\n

12-18 month vision<\/strong> \u2013 This is pretty high level and describes the kinds of scenarios we want to enable.\u00a0 It often includes some storyboard that demonstrates the value but is not intended to be either design or a feature list \u2013 it is just illustrative of the kind of experience we want to create.<\/p>\n

6 month planning<\/strong> \u2013 In this window we get more crisp about what features we are building.\u00a0 Here we work out high level cross team commitments \u2013 often our work requires coordination across multiple teams.\u00a0 It\u2019s still not design but rather agreement about what scenario we\u2019re delivering and what features support that scenario.<\/p>\n

3 week sprints<\/strong> \u2013 We generally keep a sprint schedule looking out 2-3 sprints for each feature team.\u00a0 It has a lot more detail for \u201cthis sprint\u201d than the next couple but it gives us some clarity on what is coming and when, allows the next level of dependency planning and allows us to understand what kind of progress we are making on our scenarios and balance work where we need to.\u00a0 At the end of every sprint \u2013 we deploy to production.\u00a0 Some of what we deploy may be \u201chidden\u201d behind a feature flag.\u00a0 That enables us to deploy in progress work and, where appropriate, expose it to limited sets of users to test\/give feedback.<\/p>\n

Patch Monday<\/strong> \u2013 Every Monday we are capable of deploying important but non-urgent service fixes.\u00a0 All teams know that if they have something they need to get in, there\u2019s a window every Monday.\u00a0 If there\u2019s nothing needed, we don\u2019t deploy and most of the time we don\u2019t need to.<\/p>\n

Daily hotfixes<\/strong> \u2013 On any given day, we can patch the service with a hotfix if we have any urgent service issue.\u00a0 In practice, this seems to wind up happening about once every 6 weeks (once every other sprint).\u00a0 It\u2019s usually the result of some regression that got deployed with a sprint payload but sometimes it\u2019s something else like a new load induced problem, etc.<\/p>\n

I expect as we continue to learn and mature this may evolve further.\u00a0 Maybe someday we\u2019ll break the aligned sprint model and then go to weekly deployments.\u00a0 But for now, this is working well for us and seems to be working well for the consumers of TFSPreview, so we\u2019ll keep doing it.<\/p>\n

Team Explorer Clients<\/h3>\n

Once we had started to settle on a pretty rapid cadence for service updates, we realized we were going to have another problem.\u00a0 Some of our new service features are going to require updates to the clients to really expose them in a way that works great for developers.\u00a0 This means that having a 3 week cadence for the service and a 2 year cadence for the client (or even 1 year if you count a service pack in the middle) really isn\u2019t going to work.\u00a0 So last fall, as the service cadence firmed up, we started looking at what to do about the client.<\/p>\n

It didn\u2019t take much thinking for us to realize that significant changes to the Visual Studio client every 3 weeks was probably not going to fly.\u00a0 Most customers don\u2019t want to update their clients that often.\u00a0 The quality assurance model for an on-premises release has to be more rigorous than a cloud service because fixing an issue once it\u2019s deployed is much harder.\u00a0 Etc.\u00a0 So we started looking at a model that had a few constraints:<\/p>\n

    \n
  1. Ship often enough to not hold the service evolution back.<\/li>\n
  2. Provide a bet\nter vehicle than we\u2019ve ever had before to provide responsive innovation and improvements based on customer feedback.<\/li>\n
  3. Ensure the quality of on premises updates are high.<\/li>\n
  4. Deliver updates in a way that is reasonably easy to discover, has minimal impact, and has a reasonable acquisition experience.<\/li>\n
  5. Assume not everyone will take every update so it needs to be easy to skip updates and then later pick up new ones.<\/li>\n<\/ol>\n

    We ultimately settled on quarterly updates as a reasonable trade-off between the costs of frequent updates and the lag with the service.\u00a0 However, it\u2019s clear to accomplish this, we\u2019ll need to be thoughtful about what changes we make in these quarterly updates.\u00a0 3 months is not enough time to run a full validation pass for arbitrary sets of changes.\u00a0 As such we\u2019ll have to focus mostly on changes \u201chigher in the stack\u201d to minimize the potential destabilizing impact.\u00a0 To use a ridiculous example, significant feature changes to the .NET Framework every 3 months and deploying that to the world would be a very bad idea given our current abilities.<\/p>\n

    We introduced a Visual Studio Extensions and Updates feature in VS 2010.\u00a0 As we looked at mechanisms for delivering updates to VS it looked like an appealing mechanism.\u00a0 We also considered Microsoft Update and other options but we felt the VS Update mechanism was the most appealing.\u00a0 Unfortunately, in 2010, it didn\u2019t support the full power we needed to update the breadth of components we felt we might need to update.\u00a0 Fortunately, having started to think this through last fall, we were able to pull together a plan to extend the VS Update mechanism in VS 2012 to support the power that we need.<\/p>\n

    So, the plan is to, now that VS 2012 has shipped,\u00a0 move to a quarterly update cadence for our clients.\u00a0 This won\u2019t, of course eliminate the need for us to do longer cadence \u201cmajor updates\u201d too.\u00a0 So expect major releases to continue but I\u2019m very happy to be able to provide continuing value on a regular basis.<\/p>\n

    Team Foundation Service<\/h3>\n

    Once we had locked our plans for service updates every 3 weeks and client updates every quarter, the next obvious question was about our on premises TFS server.\u00a0 It\u2019s clear that we have a large number of customers are going to continue to want to use an on premises solution \u2013 you might say that\u2019s our bread and butter.\u00a0 We also have some very good hosting partners filling needs that our Team Foundation Service doesn\u2019t address who would like to be able to provide the latest capabilities.\u00a0 How are these groups of people going to feel about waiting 2 years for features that the online service gets within 3 weeks of release?\u00a0 In fact it\u2019s worse than that.\u00a0 A few months before we released TFS 2012 we had to start locking down the churn and as a result the service was getting new features that were not in TFS 2012 *before*<\/strong> TFS 2012 even shipped.<\/p>\n

    On the other hand, in as much as it is true that not everyone wants to update their client every 3 months, it\u2019s even more true that not everyone wants to update their server every 3 months.\u00a0 Further, we don\u2019t have a clean and simple a mechanism for updating the server as we do for the client.\u00a0 It\u2019s also the case that the QA process for a mission critical server is even more involved and costly than for a client.<\/p>\n

    All this taken together, I\u2019d rather not try to update the on-premises server every 3 months.\u00a0 However, as we\u2019ve started to figure out how to put any cadence plan for the server into action, we are finding that it actually depends a great deal on what kinds of capabilities we are trying to deliver.\u00a0 So we\u2019ve ultimately landed in a place where we\u2019ve said, we aren\u2019t going to make a firm commitment to a cadence for the on-premises server.\u00a0 Instead, we\u2019ll \u201cplay it by ear\u201d.\u00a0 At this point, it\u2019s clear that we *will*<\/strong> need to update the on-premises server in our first quarterly update later this year.\u00a0 Once we\u2019ve been through one of those cycles once, we\u2019ll revisit the cadence question and see if we are in a better position to lock on a cadence or whether we continue to \u201cplay it by ear.\u201d<\/p>\n

    Conclusion<\/h3>\n

    It\u2019s a long post and I\u2019m sorry about that but I wanted to give you some flavor of the journey.\u00a0 The summary is:<\/p>\n