More Than Just the Two-Way Mirror

I've
revealed in the past how many of the Office 12 UI decisions we make are
influenced by what we learn in collaboration with our usability and research
team.

Personally, I find usability tests scary.  You have an idea, discuss it
with others, work it into a prototype and get excited about it.  You want
it to work.  And then you pull back the curtain, put the idea in front of
real people and all of a sudden you know whether the idea works for real or not. 
It's totally humbling.

It's like when you're dieting--you can feel good about the food you eat or
exercise you do, but nothing brings it into focus like stepping on the scale and
facing the result.  If the result is bad, it's back to the drawing board. 
If the result is good, you set it aside, plan to retest, and start focusing on
where else you can improve.

Over the next weeks, I'm going to write in more detail about our usability and
research efforts around the Office 12 UI.  To give you a taste, here's a
snapshot of all the special research projects we have getting started over the next
month or so.  The following bullets are excerpted from a status e-mail
message written by Tim
Briggs, one of our usability leads.

• Office 2003 Benchmark (10/19 - 10/20): 30+ people coming through
  the Office Design Lab, 10 at a time to complete tasks in previous versions. 
  We're collecting success/failure, time on task, and satisfaction so that we
  can compare to the Office 12 Benchmark later.

   

• Eye Tracking 1 & 2 (10/19 & 11/11): Using the eye tracker to give
  us better understanding of how people use and browse the current visuals so
  we can then refine and study the new visuals.  We're looking especially
  at how people use the chunk titles, the contextual tabs, and the MiniBar.

   

• Card Sort II (early Nov.): A year ago we had people organize
  commands into buckets to help us think about how to generate the tabs. 
  This time we'll give them our buckets and see how well they can sort the
  commands into them, especially while coming up with names for some really
  sticky ones.

   

• Internal Longitudinal Study (Now - Dec.): We're getting people
  from the company set to install and start using both current builds and Beta
  1.  All are outside of Office, most are from non-product groups like
  Marketing, Sales, and Legal, and will not only be sending feedback but
  participating in the Office 2003 and Office 12 Benchmarks so we can compare.

   

• The "Truman Show" (Now - Dec.): A brave soul from the "real"
  world has agreed to let us take away his Office 2000 and replace it with
  Office 12... and do site visits... and work here on campus for a while... and
  have conferences via Live Meeting with us... and let us analyze his personal
  instrumentation data... and send us a daily journal of his experience. 
  He's a local guy with a small personal business and excited
  about this challenge.  As are we.

   

• Office 12 Benchmark (Nov. - Dec.): We'll look at the whole range
  of use from OOBE ("out of box experience") with newcomers to 2 weeks/2
  months use with the internal longitudinal people.  Like the Office 2003
  Benchmark, the tasks are selected to give us some key comparisons we can see
  in the big Office Design Lab data and more personally through 1:1 standard
  lab sessions.

   

• Extended Usage Study (Now - RTM): We've started deep engagement
  with a large group of users from a local company.  We'll be rolling out
  Beta 1 to all of these people.  As this kicks off, it is the first
  major opportunity to closely monitor the rollout, training, adoption, and
  acceptance of the new UI over a long period of time.  We'll be doing
  persistent visits, monitoring instrumentation data, and collecting 1/1
  feedback.

   

• Beta Survey & Visits (Dec. - Feb.): As part of the overall
  beta plan, we'll be adding to the benchmark data with a series of site
  visits to customers to observe first-hand "use in the wild." 
  Members of the design team will be partnered up with customers so that we
  can follow up more individually starting in December.  Well also be
  following up with surveys to drill into productivity and satisfaction.

   

• This is just the known, scheduled research. There are lots of other
  efforts going on too like building up the list of instrumentation metrics,
  3rd party validation studies, continued "Send a Smile" tracking, continued
  consolidation of usability findings from Word, Excel, PowerPoint, Outlook,
  and Access, and oh so much more.

Jensen here again.  All of this is in addition to the normal ongoing
"behind two-way glass" usability tests that focus on individual mechanisms or
features.  And of course it's added to all of the beta feedback we get through newsgroups, customer
visits, e-mail (and even here in blog comments!)

As you can see, we have a lot going on as we strive to learn more and more
about the UI and how to continue improving it.  It's a large effort, but I
think it will be worth it.

I'll fill you in on the details of some of the more interesting studies in
the coming weeks.

Have a great weekend everyone!

Comments

• Anonymous
  October 21, 2005
  The comment has been removed

• Anonymous
  October 21, 2005
  Can you tell us how many usability engineers are involved in all those activities? It looks like it's a huge undertaking.

• Anonymous
  October 21, 2005
  Have you also equated a "best possible" time metric to complete a task? Based on estimates of cognitive reaction times, the time it takes to move a mouse and hit a target... I think everything you are doing is great, but I'd also be interested in how your results compare to theoretical perfection. ;-)

• Anonymous
  October 21, 2005
  I fine this facinating. I have done some work with site usability in the past, but nothing so deep.
  
  Will generalized results be made avaliable, via Microsoft Research? I find much of what I find there very useful.

• Anonymous
  October 21, 2005
  What is: "Send a Smile" tracking ?

• Anonymous
  October 25, 2005
  Interesting, Jensen.
  I'm someone who still uses Office 2000 because Windows XP broke the comments and reviewing features, and they weren't fixed in Office 2003. I don't expect them to be fixed in Office 12 either, since some of these bugs were logged and reported in the public Knowledge Base, but they haven't been fixed. So I assume they're not considered serious by the Office team, and won't be fixed. However, they're showstoppers as far as I'm concerned. I do a lot of editing and spend a lot of time writing comments for other users and reviewing their comments in my work.
  
  
  Here's a quick list of my beefs (these have all been reported at least twice directly to people in the Office team, by the way, another reason I've concluded they won't be fixed. MS knows about these and knows they're showstoppers for us, but hasn't addressed them.) Note, by the way, that we normally work in outline view, so if you try this, you should be in outline view.
  + The Delete key doesn't work on comments in the reviewing pane. Press Delete, nothing happens. I can't imagine what feature this implements, and it was recorded in the KB as a bug, but it hasn't been fixed.
  + The Backspace key doesn't work when text is selected in a comment. Select text in the reviewing pane, press Backspace, nothing happens.
  Note that Delete WILL delete if text is selected, and Backspace works if text ISN'T selected, so there's a bizarre mirror image of disfunctionality here and I gotta believe that this is a trivial fix.
  + When viewing a document in which revisions have been tracked, if you hover the mouse over deleted text with a comment on it (marked with a strikethrough) the tooltip that appears shows you the deleted text, and not the comment. This is bizarre. I can see the deleted text in my document, so showing it to me again in the tooltip (and again in the reviewing pane, if it's open) isn't useful. What I want to view is the comment about the deleted text, which typically is an editor explaining why the text was deleted. The only way to view the comment is to switch into a different view and look at the balloon (in which, by default, you can now see the comment, but not the deleted text. Who thought that up?).
  
  If you want to know how this should work, simply see how Word 2000 does it. Delete text in a file viewed in outline mode, enter a comment on top of the deleted text, and see how it is viewed on a mouse-over. Also, see how the Delete and Backspace keys work in the reviewing pane, both with and without selected text.
  
  This is a showstopper for me, because I really hate to use software that constantly reminds me that it's buggy and disfunctional. When I'm using a version of Word later than 2000, I get this reminder about every 10 minutes, and have to switch views, hunt through the (also buggy) reviewing pane, and perform various workarounds to get where I want to go. (Another, less serious isssue: what usability genius told you that they wanted the reviewing pane to hold not only comments, but every single trivial edit? Why isn't there a way to view only comments in the reviewing pane, without having to change the view of the main text to Final, thus hiding the markup to which the comments often apply?)
  

• Anonymous
  October 27, 2005
  How much wieght is put on to how new users work with the apps vs. experienced users?
  
  While everyone starts a beginner, consistent users are not going to stay that way for long.

• Anonymous
  October 28, 2005
  For that matter, how much is done with people of different age groups, abilities, and equipment?
  
  I use a pointing tablet in place of a mouse, and I would really like the things that require accurate pointing to grab something (table cell borders in Word, cell corners in Excel) prior to a drag to have wider ranges of proximity in order to trigger the grab. Perhaps this would be a preference.

• Anonymous
  April 18, 2006
  There are many ways we collect feedback about the new UI. I've detailed some of the ongoing usability...

• Anonymous
  June 28, 2008
  PingBack from http://braxton.superstoriessite.com/twowaymirrors.html

• Anonymous
  October 27, 2008
  PingBack from http://mstechnews.info/2008/10/the-office-2007-ui-bible/

• Anonymous
  June 01, 2009
  PingBack from http://indoorgrillsrecipes.info/story.php?id=5644

• Anonymous
  June 09, 2009
  PingBack from http://greenteafatburner.info/story.php?id=2086

• Anonymous
  June 09, 2009
  PingBack from http://quickdietsite.info/story.php?id=3831