Yesterday, I got a thoughtful comment from Dean on my post about Update 3.  I sat down to write a response this morning and it turned into a but of a novel (in fact, the blog says it’s too long to post as a comment ).  So I’ve turned it into a post.  Here was Dean’s comment:

Brian,

I looked at the list of fixes, you weren’t kidding when you said small.  Are those just the major ones (crashing)?  There are a whole lot of things logged on Connect that when added together really stifle productivity (many involving the editor with intellisense, syntax highlighting, etc.).  I get nervous at this point in a VS life-cycle because I fear that the bugs I care most about will once again get worked on for the next major version.  We spend money on the product and the window of time for things getting fixed is unbelievably tiny…before MS turns off the lights/closes shop until the next version.  The new update cycle is great, I just hope it is not merely SP1 in chunks…I hope you still give updates to the product all along the way up until the next version.

-Dean

And my reply…

@Dean, I try not to kid 🙂  I’m not going to swear that’s every single bug fix. Plus there will be a few more fixed between now and the final release – for instance, I just heard that we are fixing a few important Blend issues and those fixes didn’t make the RC.  It’s not an incredibly small list – I count 59 bug fixes.  I’m not sure whether to feel proud or ashamed that 34 of them are in my team 🙂  I think the answer is proud, though I suspect this is an eye of the beholder issue.  A story…

I was speaking last week with one of our test managers and he said to me…

People in shiproom don’t know what to make of us.  We bring so many more bugs to be fixed than any other team in VS that they just look at us sideways.  But, my feeling is that if a customer reports a bug, we should just fix it.  We’re doing the update anyway and there’s no point making the customer wait.

Aside – shiproom is a process we have of sharing the changes that each team is making as we get close to the end of any release that allows for peer review and feedback.  The purpose is to help slow down the churn, reduce the risk of regression and warn other teams of any changes that might impact them.

It was a proud moment for me.  I’ve spent the better part of 12 years arguing for a different way of thinking about this kind of thing.  I grew up in startups where virtually every customer felt like the difference between going out of business or not.  I spent 30% of my time on tech support and generally gave same day (or at most a couple of days) turn around on bug fixes if a customer had an issue.  I’ve always believed that trying (look, I understand that you’ll never fully succeed) to make each and every customer happy is important.  And the connection between developers and the problems customers face is an important feedback loop that forces learning over time.

I don’t want anyone to think this is a simple “motherhood and apple pie” vs ignorance issue.  There’s a legitimate debate to be had.  First, every time we fix an issue we have a non-zero probability of introducing a regression that our testing misses.  Customers tend to be super unhappy if they get a fix for one problem only to find a new problem they didn’t have before.  Further, we all know interruptions are bad – they sap productivity.  Having to stop what you are doing and investigate a customer reported issue (which often turns out to be a configuration problem), produce a fix, test it and deliver it can really reduce the overall volume of value delivered.  Further, fixing every issue any customer reports can cause you to spend an inordinate amount of time fixing issues that affect relatively few people while the bulk of your customers wait for value.  It’s not a simple trade-off and neither extreme is the right answer.  It’s a balance and it all comes down to the values to apply to weigh that balance.

As for the instability introduced by regressions, I argue that some regression rate is acceptable as long as your time to repair is sufficiently short.  While a customer will be frustrated by getting a new bug along with their fix to the last, as long as it doesn’t happen too often and you fix the new bug quickly, the net result of fixing people’s issues more responsively is a win.  Reasonable minds can disagree on what rate is considered acceptable.  The lower you want to drive the rate though, the less churn you can tolerate and the less frequently you can release.  Generally your tolerance also varies by the kind of component you are working on, the ease of deploying updates, and many other factors.

Now, again, before someone jumps to calling me a hypocrite, let’s talk about Team Project Rename.  There’s an “issue” that has existed for a long time and we’ve done nothing about it (right Allen?).  Could we?  Of course, it’s just software – anything can be done.  So that must mean we have decided not to despite it being one of the top customer generated requests.  Brian, you’re a hypocrite.  I like to think I’m not but you can judge that for yourself.  As I said, it’s a balance.  This is probably one of the things I hang my head in shame about.  We made some decisions many many years ago that, in retrospect, I would do differently.  The ramification of those decisions are that Team Project rename is hard – it shouldn’t be, but it is.  We’ve costed it a couple of different times over the past several years and it always comes back as months and months of work with high regression probabilities.  That doesn’t mean we’re ignoring it.  I still very much want to do it but we’re taking a longer road to get there.  For a while we just decided to “live” with the problem.  Now we have a plan but it calls for making changes to “undo” some of those decisions we made many years ago that makes it hard.  Once we get that work done, we can do rename.  All I can do is apologize for how long it is taking.

Back more to your point Dean.  Yes, Update 3 is likely the last of the updates to the VS 2012 line.  Of course, we’ll still continue to fix any critical issues people find but we are winding it down and focusing on VS V.Next.  I’d like to think many of the issues you refer to will get addressed there and, if not, I hope we’ll get to them in the V.Next update train.  Referring to the above – Update 3 is the update where we slow down the churn, address remaining high impact customer issues and any regressions introduced in Update 2.

When you say the window is “unbelievably tiny”, I don’t think I agree.  Of course we tried to get as much customer value and feedback incorporated into 2012 RTM as we could and then followed with 3 updates over a period of 9-10 months in which we delivered, in the aggregate, a tremendous amount of stuff – much of which was directly driven by customer feedback (e.g. Desktop Express SKU, C++ XP targeting, Kanban support, Blue theme, and dozens of more significant improvements – and, hundreds of bug fixes).  While I can understand it’s frustrating that we didn’t get everything – or even all the most popular ones, we did make a lot of progress – and we ‘ll keep making progress in the context of V.Next.

I also don’t think it’s “SP1 in chunks”.  The kinds of changes we’ve put into the updates go FAR beyond what we would have historically included in a Service Pack.  Service Packs had an “aura” that they only contain bug fixes and while that was never strictly true – any time someone proposed a Service Pack change that didn’t smell like a bug fix, there was a lot of justification that had to be done.  One of the fundamental mindset changes with the move from “Service Packs” to “Updates” has been that the primary value of Updates is new value – and sure we’ll fix a lot of bugs too, but that’s not the focus.  Read my posts on the updates and you’ll that generally the bug fixes are a footnote.  They are all about the cool new capabilities we are enabling.

I’ve exposed a lot of flank here – so I suspect it may generate a lively conversation.  But, hopefully it sheds a little light on how I think about it (of course, only one person in a large company – so don’t construe this as any kind of official policy).  As always, I’m happy to engage in healthy debate and learn from customers and from mistakes.

Thanks as always for listening,

Brian