Share via

A Blog's Life: A Windows Vista Test Manager Addresses Bug Count

  • Article

You can be forgiven for having missed the fanfare around the kick-off of Windows Test Manager Paul Donnelly's blog, entitled Customer feedback makes for a better Windows ... because there wasn't any. Paul's initial post is unassuming but provocative, as he responds to a Longhorn Blogs article by Robert McLaws entitled Windows Vista Bug Reports: An Analysis, where Robert analyzes the team's incorporation of customer feedback into the Windows Vista development process.

In "About all those bugs, Robert...," as Paul puts it, it's fair to take into consideration two points that Paul makes in an attempt to clarify the team's handling of customer feedback. First, Paul states,

About the increasing quantity of bug reports – what is not factored in is the quantity of people we are adding to our programs. We are constantly adding new people to our programs to get more feedback. That coupled with the sheer interest generated as we approach the final stages tends to account for the uptick.

Thus, it's fair to say that the ratio of bugs per customer is steadily dropping as we incorporate more customers into the testing process while continuing to resolve issues. This should be interpreted as a good thing, in that each individual customer is discovering fewer and fewer issues. It shows that we're improving the product with each successive version and that our customers are playing a vital role in that.

Second, and arguably of even greater significance, is Paul's comment about the gravity of the issues being reported of late:

What is significant is that the severity [of] reported bugs is dropping. We’ve moved from seeing the serious issues you see in a Beta 1 to a very usable system that generates more ‘this doesn’t work quite right’ issues at Beta 2.  

Again in this case, this should be construed as a very positive trend and evidence of the impact our customer-engagement programs such as TAP (Technology Adoption Program) and CPP (Customer Preview Program) continue to have on Windows Vista.

The funny thing is that this is to my knowledge the first time that one of our customers has gone so far as to calculate the number of issues identified by customers over time and its correlated effect on product quality. While you should not expect us at MS to publish such information publicly, you can nevertheless take this as evidence that our engineering teams are well aware of the effect customer feedback has on identifying and resolving bugs in Windows Vista.

Comments

  • Anonymous
    January 01, 2003
    A couple of days ago, an article was published over at TechWeb, stating that Vista will ship in late...
  • Anonymous
    July 13, 2006
    Why exactly can't we expect you at MS to publish such information publicly?  That sounds like a very old-guard Microsoft stance vs. a more open MS.  I'm very curious how publishing bug-fix stats could be anything but a good thing.
  • Anonymous
    July 14, 2006
    Maybe their focus isn't to publish how many bugs get fixed etc - maybe their focus is to fix the bugs that do get reported.

    I mean Robert made a GREAT post with GREAT information but as a user and a tester I could care less on stats. I want to submit bugs and see they get fixed. I want to see progress and progress isn't seeing bug stats - progress is getting a new build that stops using 50% of my memory all the time. Or have DWM work a bit better. Or a new theme for non-Aero users.
  • Anonymous
    July 14, 2006
    The comment has been removed
  • Anonymous
    July 15, 2006
    The comment has been removed
  • Anonymous
    July 15, 2006
    The comment has been removed
  • Anonymous
    July 16, 2006
    The comment has been removed
  • Anonymous
    July 16, 2006
    I think that when you post a reply it should not be censored, but unfortunately that seems to be the case.
  • Anonymous
    July 16, 2006
    I was unaware of any censoring going on with comments of this blog.

    But if you wish to continue this discussion without any possibility of censorship, feel free to email me at brandonleblanc[at]hotmail.com. I'd be more than happy to continue the discussion.
  • Anonymous
    July 16, 2006
    The comment has been removed
  • Anonymous
    July 16, 2006
    The comment has been removed
  • Anonymous
    July 17, 2006
    HKLM -- while anonymous comments are disabled, as is typical of many blogs, I do not edit comments on this blog.  Further, I'm the only one with access to the blog's back-end editorial functions.  

    It really doesn't do me much good to create an official MS blog and then to edit the comments, as those submitting comments will figure this out in short order, publicly destroy my credibility and then no one will visit the blog.  As you can see based on the number of comments received, that latter point in not the case, which supports my statement that comments are not edited (or otherwise censored).

    You're free to email me if you have further to say on this particular issue.

    -- Nick
  • Anonymous
    July 19, 2006
    The comment has been removed
  • Anonymous
    July 19, 2006
    The comment has been removed