Why exams mean nothing out of context

November 23rd, 2010 by Sabrina

One of the advantages of remote usability testing is that you are using participants in their real context.

Before we started Webnographer, James and I attended a lecture given by Robert Sternberg where he told the story about the maths ability of Brazilian street kids living in the in the favelas of Recife. This story helped both of us realise the importance of carrying out usability tests in context.

Three researchers (see: Carraher, Carraher, and Schliemann 1985) carried out research with children aged 9 to 15.  These kids had dropped out of school, and were selling sun screen, and chewing gum on the streets. The researchers worked out that they could set the kids questions by purchasing goods off them. For example, 1,000 minus 300  is the same as giving the kid a 1,000 Cruzeiros note for a product that costs 300 Cruzeiros. Multiplication can be done by asking the kids how much 3 of a product would cost.

In these tests the Brazilian street kids scored 98%. But when they were put into a formalised test setting, and asked instead of how much would 3 apples cost or what 3×9 is, the kids performance dropped to just 37%.

What is scary is that the researchers later tested middle class children in a private school. These kids did very well in the formal exam. But when they had to do transactions with real money in the street, using the same maths, they failed in being able to do the transactions.

This is one example of context and how it affects behaviour. We see it again and again with remote usability tests, that the context of the user really affects their performance. Test people where they normally use a website, and test people in an artificial environment like a lab, and behaviour will be very different.

4 Responses to “Why exams mean nothing out of context”

  1. [...] This post was mentioned on Twitter by Rahul, eBot, webnographer, steve sanderson, Hacker News and others. Hacker News said: Why exams mean nothing out of context: http://bit.ly/idaBb6 Comments: http://bit.ly/eakkPh [...]

  2. Ruben says:

    Well, maybe the problem is not just the behavioural context but also the lack of practice of the mind flexibility skills. The context may be important to understand why children use in a more efficient way one or another “mathematical rule style” but I think that for itself cannot explain the bad results on formal or informal tests.

    In Portugal gipsy children go mainly to school to learn numbers and mathematics operations. They usually perform quite well in maths and badly in verbal activities, and this can be observed throughout their life. Their context may explain the reasons for some of these facts but do not totally explains why they perform well in both formal and informal test.

  3. [...] see it again and again with remote usability tests, that the context of the user really affects their performance. Test people where they normally use a website, and [...]