The Evaluability Hypothesis

By Rich Shaw, Research & Development

Is it a good idea to ask people to rate products all at once?

I’ve just finished working on a project where we asked people how likely they would be to purchase a number of different products - a pretty standard market research thing to do. However I was worried - not staying up at night worried, just worried enough to do a bit of thinking worried - about how rating all the products together, rather than separately, would affect the results of my purchase question. I’ve done a bit of personal research and haven’t developed any concrete answers, but came across an interesting concept...

The evaluability hypothesis

The evaluability hypothesis was introduced by psychologist Christopher Hsee (1996) and is stated as follows:

“When two stimulus options involve a trade-off between a hard-to-evaluate attribute and a easy-to-evaluate attribute, the hard-to-evaluate attribute has a lesser impact in a separate evaluation than in a joint evaluation, and the easy-to-evaluate-attribute has a greater impact.”

What on earth does that mean?

The evaluability hypothesis breaks (product) attributes down into two types - easy-to-evaluate independently, and hard-to-evaluate-independently. Easy-to-evaluate attributes can be assessed by a person without needing to compare it to something else. Hard-to-evaluate attributes are difficult to assess without comparing themto something else.

If you have products which have a easy-to-evaluate attribute and a hard-to-evaluate attribute, when the products are evaluated independently people will place more emphasis on the easy-to-evaluate attribute. If the products are evaluated together then people will place more emphasis on the hard-to-evaluate attribute.

Seriously, what are you going on about?

An example might help explain where this is leading! Imagine we’re doing some research which involves looking at how much people are willing to pay for two cameras. We ask people how much they would be willing to pay for each of the camera’s assuming they had a budget of £10 - £150 to spend on a camera.

 
Mega-pixels
Defects

Camera 1
5
None

Camera 2
10.2
Scuff on the case otherwise as new

If the cameras were rated separately in two conditions you would expect that Camera 1 would get the highest mean value (in terms of how much people would be willing to pay) because people may not be sure how good 5 mega-pixels is (hard-to-evaluate attribute). But they know no defects is good (easy-to-evaluate attribute). If the cameras where rated together you would expect Camera 2 to get the higher rating because people know no defects is good (easy-to-evaluate attribute) and that in comparison 10.2 mega-pixels is better than 5 mega-pixels (hard-to-evaluate attribute). 

So, what does it all mean?

We can take out a couple of things out of the work of Hsee. When asking people to rate products or concepts think carefully about what attributes people are rating them on. Are the attributes easy-to-evaluate or hard-to-evaluate and how could this affect the data you collect? This work could also have a bearing on the recommendations we give to clients. For example should a product be advertised alone or in comparison with other products?

References

Christopher K. Hsee, The Evaluability Hypothesis: An Explanation for Preference Reversals between Joint and Separate Evaluations of Alternatives, Organizational Behavior and Human Decision Processes, Volume 67, Issue 3, September 1996, Pages 247-257, ISSN 0749-5978, DOI: 10.1006/obhd.1996.0077.
(http://www.sciencedirect.com/science/article/B6WP2-45N426D-8/2/b00fa36d56173e883d2b163ddaabaf13)

Ice cream image taken from lesswrong, 'a community blog devoted to refining the art of human rationality': http://lesswrong.com/lw/lh/evaluability_and_cheap_holiday_shopping/