1000 Card Pick-Up

Today, we got back a pile of data from a recent card sort exercise.

We brought in 17 Word users and 9 Excel users and gave them a huge stack of virtual "cards" containing the name of a command and a short description of what the command does.  They were also given the proposed names of Ribbon tabs (both core tabs and contextual tabs).  The subjects were asked to stack the commands where they think they belong solely based on the names of the tabs and the commands.

Beyond that, the subjects weren't given any other instructions and they weren't allowed to see Office 12 or the new UI at all.

What we got back were these wonderful huge Excel spreadsheets, each one containing 20 worksheets chock full of raw data.  I haven't had time to go through it in detail yet, and, in fact, the usability team itself hasn't had time to fully analyze it either.  It's hot off the presses.  But still, I can't resist looking it over and making some rush judgments.

These are not the cards we use in usability tests

There are always some things in usability data that make you scratch your head.  You think to yourself "really, someone thinks Check Spelling belongs on the Format Picture tab?"   Do you ignore that data?  Assume there's a bug in the test?  Chalk it up to disinterest on the part of the subject?  Or take it to heart, following the usability principle "the software's wrong, not the user."

Overall, tab categories were scored based on two criteria:

• Number of "Errors of Omission": how many commands should have been placed in a tab but were instead placed somewhere else.
   
• Number of "Errors of Inclusion": how many commands people placed in a tab that were supposed to be somewhere else.

Some tabs scored marvelously (above 80% correct with few errors of inclusion), a few did less well (less than 50% correct.)

Interestingly, the first tab of each application, which is designed as a kind of efficient clearinghouse for the most-used commands, scores poorly in a blind test because the current tab names ("Write" for instance in Word) are not descriptive.  I tend to take that data point a bit less seriously because we've seen people be successful using the first tab over and over.  Because it feels like "home", it seems to matter a bit less what we name it.  (There was a time when we thought about naming the first tab "Home" or "Start" or even the program name.)

So, there's tons of data to look through here, and I can't wait to see what we learn from it.

What are the actions we could take based on the card sort data?  There are at least three different possibilities.  We could decide that a tab name is not descriptive enough and try out different names for it.  Or, we could decide that our organization isn't fitting with the way people think and shuffle things around accordingly.  Or, we could use other kinds of tests to explore a particular aspect of the results from a different angle, trying to validate or invalidate the need to take action.

Honestly, I'm feeling pretty good about most of our organization although we do continue to move things around.  Finalizing the tab names will be a meaningful process, and I'm already feeling a bit of the fear of commitment.  I want to live up to the legacy of the people who came up with "File, Edit, View, Insert, Format, Tools, ..." and make sure that we ship the best names and feature organization possible.

Comments

• Anonymous
  November 10, 2005
  So 26 people represent 400,000,000 users.
  
  I didn't pay attention closely enough in my two mandatory semesters of Statistics to remember much about "confidence" / "degree of certainty".
  
  Is there anything else you're willing/able to share about how they were picked or why that many? i.e. Was this number low because of the amount of data generated, because of how they were picked/screened, or something else?
  
  Maybe I'm the only one curious about it :-)
  
• Anonymous
  November 10, 2005
  The comment has been removed
• Anonymous
  November 10, 2005
  The comment has been removed
• Anonymous
  November 10, 2005
  @Michael Zuschlag: Ha! Especially since the Write tab is on the leftmost side. Yep, that'll make supporting Mom a lot harder! :-)
• Anonymous
  November 10, 2005
  Ryan,
  
  I'm just not sure that "I, Nephi..." would have the same effect as "In the beginning..."
  
  :)
• Anonymous
  November 10, 2005
  Chris: When I took statistics, I remember my professor saying that 30 is infinity. That probably only applies to normal distributions, but by that definition they're half way to infinity for Word and a third of the way for Excel.
  
  Ryan: Even if Office was localized into Utahn, I have to agree with Kim that "In the beginning" would have a more appropriate meaning than "I, Nephi". But to keep it agnostic, we could just call it something like "Call me Ishmael".
• Anonymous
  November 10, 2005
  The comment has been removed
• Anonymous
  November 10, 2005
  The comment has been removed
• Anonymous
  November 10, 2005
  I would agree with the idea of calling it "Main" consistently through all the apps.
• Anonymous
  November 10, 2005
  Come to think of it, I can see the point about the "Write" and "Sheet" tab.
  
  Confusion. A play in two Acts.
  
  Act 1:
  
  John: Click the Write tab...
  Sally: Right-click which tab?
  John: No, no, left-click the Write Tab.
  Sally: Huh? How do I know which is the right tab?
  John: No, the wwwwright tab.
  Sally: Whataya mean, the tab ON the right??
  John: -click- bzzzzzzz.....
  
  Act 2:
  
  Linda: Now on the sheet tab, we're going to...
  Doug: Which sheet tab?
  Linda: What?
  Doug: Which sheet tab? Sheet1, 2, or 3?
  Linda: No, the sheet tab at the top.
  Doug: WHICH SH*TTY TAB??
  
  Curtain.
  Applause...
  
  Seriously, though, if your useability studies tell you that users understand these tabs with those names, then that's fine.
  
  I don't know about getting a standard name: MAIN. Few of the other tabs will be standard. And if the FIRST TWO are the same (Main, Insert), people might EXPECT the others to be the same, too. Maybe making them distinct is a good idea.
• Anonymous
  November 10, 2005
  Office 12 invites are going out:
  Microsoft has just sent out the first wave of "Office 12" technical beta program invitations. In the e-mail received by many of our members here at Neowin, Microsoft revealed that Office 12 Beta 1 is scheduled for release within the next 2-3 weeks.
  
  "Thank you for completing our nomination survey sometime within the past several weeks, and letting us know about your interest in participating in the Microsoft® Office "12" Technical Beta program. We are pleased to announce that your request for participation in the Office "12" Technical Beta has been approved. You will be provided with access to the Office "12" Beta software, documentation, and support within the next 2-3 weeks. But we wanted to let you know now that you have been accepted, so that you can make any preparations that you need."
  
  Read the rest here:
  http://www.neowin.net/comments.php?category=main&id=31392
• Anonymous
  November 10, 2005
  Jensen said:
  > someone thinks Check Spelling belongs on the Format Picture tab
  > Do you ignore that data?
  > Assume there's a bug in the test?
  > Chalk it up to disinterest on the part of the subject?
  > Or take it to heart
  Another possibility is that you & the user have different understandings of what the terms "Check Spelling" and "Format Picture" actually mean. So the user is answering a different question, to the one you think you asked. Eg. the user thinks "Check" means cheque as in banking, so they put "check spelling" under the mailmerge menu, because they often generate bank cheques using a mailmerge process. Perhaps a silly example, but you get what I mean.
• Anonymous
  November 10, 2005
  TC:
  
  It's possible, and the suggestion is well-taken. The participants are given a short setence of paragraph for each command telling what it does, so I'm less apt to blame that in this case.
  
  For instance, the Check Spelling card might have said "Check the spelling of all the text in the document to make sure that they are spelled correctly."
• Anonymous
  November 11, 2005
  Are there follow-up interviews with the subjects on the reasoning behind their choices? It would be interesting to hear their explanations.
• Anonymous
  November 11, 2005
  Maybe the "Write" tab should be "Type" or "Compose" since that is what you are doing in a document.
• Anonymous
  November 11, 2005
  Kawigi, you’re thinking of the sample size at which a t distribution is pretty much the same as a z (normal) distribution. However, the sampling distribution in this case is binomial (a command is either correctly classified or not), and, in any case, the standard error of a sampling distribution continues to shrink linearly with the square root of the sample size.
  
  Chris C., if 14 out of the 17 Word users correctly classify a given command (82%), the 95% confidence interval is between 59% and 96%, so yeah, a sample size of 17 is only going to give you a rough ballpark.
  
  BTW, if you don’t feel like working out the binomial probabilities (in Excel, of course!), a quick-and-dirty “worse case” estimate of the confidence interval for binomial data can be had by taking the inverse of the square root the sample size. So figure +/- 24% for any Word statistics, and +/- 33% for Excel.
  
• Anonymous
  November 11, 2005
  Michael:
  
  Thanks for the analysis. The math is interesting.
• Anonymous
  March 15, 2008
  PingBack from http://babynamesongsblog.info/jensen-harris-an-office-user-interface-blog-1000-card-pick-up/