Presenting findings about emotional responses

I like to experiment with different ways to present the findings from design research so they are engaging for my clients. I would be interested to get your thoughts on my recent experiment of combining Microsoft’s Product Reaction Cards with tag clouds.

Some background

For a recent piece of research, I was asked to capture data on the participants emotional response to a proposed redesign, and how that response compared with competitor sites.

As this is slightly out of my usability comfort zone – I scratched my head a bit and fell back on my emotional response stand-by “Product Reaction Cards”. The exercise I came up with was as follows:

  1. I showed the participants the homepage of a website and asked them to spend 1 minute having a look at it and considering there first impressions of the page.
  2. I then gave the participants a checklist containing all of the ‘product reaction card’ words
  3. I asked the participants to tick as many or as few of the words that seemed to resonate with their first impression of the page.
  4. I repeated steps 1-3 with a number of other sites.

The entire exercise took around 15 minutes – obviously I had a lot of other objectives to cover as well!

The results

After completing all of the research sessions, I collated the data in Excel and looked at how many times each word had been ticked. At this stage the data seemed interesting, but completely unengaging.

I considered turning the data in simple bar graphs, but decided instead to experience with presenting the data as tag clouds. Also I find presenting qualitative data in the form of quantitative data can make me feel a bit uncomfortable.

The result was a number of charts like the following:

Tag cloud

These enabled my client to quickly get a sense of the most common reactions to each sites.

I think this works as a way of presenting this kind of qualitative data, what do you think?

Advertisements

6 responses to “Presenting findings about emotional responses

  1. But these *are* quantitative data. Your data are the frequencies of ticks for your words. That’s a quantity.

    The problems with the tag cloud over a bar chart are: (1) The scale of the word is confounded with the size of the word, so longer words carry more visual weight even when the frequencies are the same. (2) It’s not apparent to the viewer if the font size is scaled linearly with the frequency or by the square root of the frequency. If you scale the font size linearly with the frequency then the visual impact is equal to the square of the frequency –words with somewhat higher frequency will appear to have had a disproportionately greater reaction (it’s a form of lying with statistics). Scaling the font by the square root of the frequency corrects for this, and maybe you did that, but the viewer can’t tell what you did and may misinterpret the graphic one way or the other. (3) Comparing one site to another is difficult, especially on a specific word because the by-pixel positions of the words jump around in different tag clouds. There are also no consistent landmarks to orient the viewer between tag clouds. It may not even be apparent that the same word lists were used for both sites. (4) There’s no facility for grid lines or other reference marks to allow precise comparisons between words within a tag cloud. Is Useful the same size as Busy or slightly smaller? I can’t tell. (5) The viewer can’t read off a precise value of a particular word. Does 14 point mean 10 ticks or 15? Or 150? (6) While it is easy to quickly pick out the highest frequency words, it’s very difficult to pick out (or even read) the lowest frequency words. Sometimes the real lesson in human metrics is what participants *didn’t* do. (7) It doesn’t scale well to richer and more sophisticated data representations such as found in stacked bar graphs or range bars.

    And the advantage of the tag cloud is… what exactly? I don’t think it saves pixels. Have you checked out Tufte?

  2. simplerisbetter

    Michael,

    Thanks for your comments – however feisty they are! There is nothing worse than not being read at all, and next to that there is nothing worse than stimulating no responses at all. So thanks!

    First off I think some clarification is in order. I conduct qualitative research, i.e. research that involves insights that can inform the design of products/sites, but that are not statistically valid. You are correct to call me on this – I should have said “not statistically valid data” rather than “qualitative data”.

    As for your comments on the tag cloud visualisation of findings…

    From a scientific/mathematic perspective I am sure you are absolutely bang on, but I’m not sure that that is the most engaging way to present information about emotional responses/desirability – in fact I am definite, I don’t believe bar charts are the best way to present such information.

    One of the most important things about communicating information is knowing your audience. This should influence the format and style in which information is communicated. Obviously in this I have a distinct advantage over you because I knew who I was presenting this too.

    I agree with you the tag cloud presentation is ambiguous, but this is actually one of the reasons I like it as a way of presenting data collected during qualitative research.

    I appreciate tag clouds are zeitgeist-y, but then again it is for presenting findings today, not to be left as a piece of research gathering dust on a shelf!

    I don’t think whether BUSY or USEFUL are different font sizes actually matters – what matters is that you can quickly and easily see that these are 2 words that were the most common reaction. I agree that it is difficult to pick out details such as the lowest frequency words, but I was giving the client a snapshot, a feel for the results.

    BTW: I also provided the client the raw data – and I asked him this afternoon whether he’d looked at it. His direct quote to me was:

    “I’m so glad you presented the findings the way you did. It made it so much more engaging than it could have been”.

    As for Tufte – information design was a key aspect of my university degree (many moons ago) – and I’ve long been an admirer of his. Have you?

    Thanks again for your comments

    Iain

  3. I guess it depends what you mean by engaging. From a tag cloud, a viewer can say, “Well, this word and that word have high frequencies, those three have moderate frequencies, and I can’t really read the rest. Okay, next slide.” They can’t get a whole lot of interest out of it, and therefore it doesn’t hold their attention. The cloud is only one step better than a bulleted list of “Top two words” and “Second highest three words.” But maybe that’s all you wanted.

    I wouldn’t say that everything Tufte says is Truth, but specifically I was thinking of his main argument from Visual Display of Quantitative Information that excellent graphics give the viewers the greatest number of ideas in the least time and space, and his criticism of graphics that employ designs just to be cool-looking without improving communication. My idea of a great graphic is one where the main point jumps out at you, but the more you look, the more details and precision you see. IMO, that’s something the bar graph gives better than the cloud. Regarding being engaging, I was also reminded of Tufte saying, “If statistics are boring, you’ve got the wrong numbers.”

    I’m puzzled by your statement that these numbers are not “statistically valid.” If the frequencies cannot be trusted (e.g., the sample size is so small there’s a reasonable probability that the high frequency words are actually moderate or low, or vice versa), then why present them at all to your client? If the numbers are junk, then calling them “qualitative” doesn’t make them less junky. And if the frequencies are _semi-trustworthy_, shouldn’t that be indicated in your graphic (e.g., by using a bar graph with error bars)?

    I don’t mean to be feisty :-).

  4. Pingback: offline.pl » Karty reakcji w testach użyteczności

  5. Iain,
    I can see where you’re coming from, and I think if you wanted to communicate the emotional response to a single design then the cloud might be good.
    But I also take Michael’s point that trying to compare the clouds from two different designs would be difficult. Some of the problems Michael pointed might contribute to someone interpreting one design as “better” than another because of the layout of the cloud.
    In summary: if I had to compare designs I would probably use a nice graph, but for just one design I might use the cloud.
    Either way, thanks for sharing your experience.
    Pat

  6. I really like this approach to displaying the results of product selection tags.

    If you’re looking for a fast and easy way to make tagclouds you might want to check out http://www.wordle.net 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s