Posts Tagged ‘Webnographer’

What usability experts can learn from cockroaches

Thursday, April 21st, 2011

James recently gave a talk at the Polish IA Summit on How Context Effects Behaviour. One of the stories from his presentation outlines that there is a lot that usability experts can learn from cockroaches. The background of the story comes from Dan Ariely’s latest book “The Upside of Irrationality”.

Usability experts often forget the affect context has on human behaviour. Most usability evaluations are still carried out in labs with an evaluator peering over the participants shoulder. And the affect that the evaluator may have on the participant’s performance is often forgotten about.

But even simple creatures such as cockroaches are affected by another cockroach watching them while carrying out a task.

In his book, Dan describes an experiment that was carried out by 3 scientists in 1969. Zajonc, Heingartner, and Herman evaluated how fast cockroaches could accomplish a task in 2 different test conditions. In the first, they were alone. No other cockroaches were around. In the second, the social condition, they had an audience. Another cockroach was watching them through a Plexiglas window that allowed both creature to see and smell one another, but that did not allow any direct contact.

What the experiment found was that, carrying out an easy task – simply walking down the length of a corridor – the performance of the cockroach in the social condition increased. They walked down the corridor faster.

However, when carrying out a difficult task – navigating through a complex maze – the cockroach in the social condition performed worse.

This is an example how context affects behaviour. It is not just a trait that affects humans, but even for cockroaches’ behaviour is affected by changing external conditions.

Of course Dan Ariely does not generalise the findings from research with cockroaches to humans. He carried out many other experiments that looked at how bonuses affect behaviour. Bonuses change the social context too. People plan in advance what they will do with their bonus – go on holiday, buy a car, a present for their partner.

What Dan found in his experiments was that bonuses are a tricky business to get right. If the bonus is too great, the pressure on the person becomes too great and they actually perform worse.

Changing the social context in which we evaluate websites and design, will change the way people behave.

Lab testing has value to get in depth interviews with participants. However, for evaluating people’s real behaviour on a website as well as their performance, usability experts should look to compliment their tool box with methods such as our un-moderated remote usability testing tool Webnographer, where people participate from their natural environment without an evaluator being present.

References:

Dan Ariely “The upside of irrationality – The unexpected Benefits of Defying Logic at Work and at Home,” (2010): 17-52

Robert Zajonc, Alexander Heingartner, and Edward Herman, “Social Enhanacement and Impairement of performance in the Cockraoch,” Journal of Personality and Social Psychology 13, no. 2 (1969): 83-92

Related posts:

Why exams mean nothing out of context

Client Testimonial: Exceptional and fascinating to work with

Tuesday, March 16th, 2010


It is a great feeling when you have worked hard with a client and they then recommend you on a public mailing list. Simon, who is a User Experience consultant at bSkyb gave this fantastic feedback about us on the IXDA mailing list:

I have just completed a large project for bSkyb. We used Webnographer for the remote testing. http://www.webnographer.com

They were exceptional as their value is in understanding the core nature of the research, translating this into a suitable test, running the test and then reporting back in a way that sheds light on the fundamental causes of the issues. A lot of these services are dumb online questionnaires that can do more harm than good.

James Page and Sabrina Mach have been fascinating to work with. Their background in behaviour science, engineering, anthropology, ethnography, UCD, psychology is astounding. Their broad range of skills makes a huge difference to the quality of work produced.

I should say that I am no affiliated in any way whatsoever with Webnographer. I am a freelance UxC that knows Webnographer from the days I worked at Flow Interactive.

Good luck with your research.

Yours,

Simon Johnson
User Experience Consultant at bSkyb

Is All Remote Usability Testing The Same?

Tuesday, February 24th, 2009

Every time we talk about remote usability testing, many people have many different ideas of what remote testing is.  And there are many different remote testing methods.

Rex Hartson in 1996, defined remote usability testing as the method “wherein the evaluator, performing observation and analysis, is separated in space and/or time from the user”.

Two simple distinguishing factors, which are now commonly accepted, are that remote testing falls into two main categories:

  1. Synchronous / moderated remote testing, where the evaluator has direct communication with the participant during the evaluation. This is normally carried out using screen sharing software, or talking over the phone.
  2. Asynchronous / automated remote testing/ automated remote testing, where the evaluator has no contact with test participants. Asynchronous remote testing is the method used by Webnographer.

Asynchronous remote testing falls into three sub categories.

The simplest is analog remote testing, such as critical incident reporting, where the participant is sent a paper questionnaire which is filled in after carrying out the test. While this may seem like an easy method for evaluators, this is a method which can be very time consuming for the participant.

A bit more advanced are task-based online questionnaires, which are online tools that query the user for feedback after carrying out the task. Yet, those do not track any interaction on the page which makes it difficult to infer why problems are occurred. They therefore can only offer summative results such as success rates, time on task and satisfaction.

The most advanced tools use the CARUT methodology (Combined Asynchronous Remote Testing methodology), which combine logs of the user interaction during the task (hovering, clicks, key press, etc.) with questionnaires, which record rich qualitative feedback. This allows inferences about where and why problems occurred and offers both summative and formative results. Webnographer uses the CARUT methodology.

Yet, a challenge with most of the remote tracking tools which use the CARUT methodology is that they require the test participant to download software for it to work. We have found most people, worried about viruses and trojans, do not want to download software, especially one that tracks them. The other disadvantage of downloaded software is that it normally only works with certain types of browsers and operating systems. We have found that being able to work with different combinations of computers and browsers is important as we found the test participant’s configuration of their computers has a major impact on how they use a site.

Webnographer does not hold the above challenges. It works with nearly all computers, and with the most browsers. The user does not have to download anything and the website does not need to be modified. Webnographer gives therefore full freedom to carry out any test, including tests on your competitor’s website for competitive analysis.

Webnographer – Where we are at!

Tuesday, February 17th, 2009

This is an update of where we are with Webnographer.

We are developing Webnographer using Agile Project Management Techniques. Agile builds the project in very small iterations of between 1 and 4 weeks. At the end of each iteration the software should be working. To give an example with Webnographer in our first iteration the whole system was usable from start to end. I.e. we could build a test, get participants to test a website, and analyse the results.

Not only did we want to just build a tool that could give summative results, for example how long a user takes on a task. But we wanted to build a remote testing tool that could be used for formative testing as well. Webnographer enables this by capturing users interactions on the page, as well as qualitative and quantitative questionnaire data. This means that Webnographer does not just report the state of a system (as in summative evaluations), but also provides insight into where, how and why user errors occur (as in formative evaluations).

So how could we build complex usability testing software in a week? We didn’t. When we started, the building of the test and the analysis of the results had to be done by hand. Over a number of iterations what had to be done by hand got automated.

The Agile technique is a revolt against the traditional waterfall approach. Under the old fashioned waterfall approach the business specifies exactly the software that will be built, the developers build it, and then it gets tested, and hopefully released. The challenge with the waterfall approach is that it often led to software projects running over time and budget.

With Agile each iteration can either add features or rework what has been built before. This has the advantage for developing Webnographer, that as we get feedback from each of the tests that we run with clients, the results feed back into each iteration and add to an improvement of the tool.

Up until now, we have mainly focused on the design of the test set up. The test set up was the most time consuming part and easiest to automate for Webnographer. Before it had its simple interface, to create a test each one had to be hand crafted, taking half a day. Modification of a test was hard, time consuming, and error prone. So we have focused on developing an easy to use interface for the test design. This is now complete.

The next iterations will focus on the analysis part of the product. The driving force is that we want to show people actionable results in an easy to understand way. Sample reports are to follow soon.

Why I started Webnographer

Wednesday, December 10th, 2008

When I was 11 years old my mother gave me for my birthday an Acorn Atom (One of the first personal computers). That was 28 years ago. Since that day I have been the one to rescue most of the technology problems in my family. Luckily when my brothers got older they took over some of the rescue calls.

This computer led me to start being intrested in how people use computers.

This computer led me to start being interested in how people use computers.

Most of the technology challenges my family faced have been them struggling with usability issues.  Later when I was paid to write programs I wanted to make software easy to use, because I wanted people to use the software. When I started Deckchair.com with Bob Geldof our whole ethos was to make booking flights less frustrating, and simpler. I think we succeeded on the whole. Before the Internet flight booking engines, it took months for somebody to learn how to book a flight using a computer. When online bookings became available consumers where able to do it themselves without training. How many people still use a Travel Agent now?

Many Usability experts blame the programmer for the Usability Challenge. If you read Alan Coopers book The Inmates Are Running the Asylum you will find a rant in how programmers are to blame. He uses a very narrow stereotype of the programmer to illustrate his argument. The second issue I have with Coopers blame game, is that he does not acknowledge that most of the innovation in usability has actually come from programmers. Jef Raskin and the apple Mac, Douglas Engelbart and the GUI, the mouse, and hypertext. Or even more recent movements like Web 2.0 the innovations have mainly been led from the programming field.

Most programmers I have worked with are as frustrated by usability as much as anybody else, maybe even more so because they have to use computers all the time, and as Paul Graham, the creator of the first web based application said :-

Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.

I believe that we can start solving the challenge of Usability once everybody starts understanding what the issues are. Very few people design systems not to be usable. If there is an usability issue, it is there because the programmer or designer overlooked something, or did not expect the user to behave in that way, or did not realise the importance of the issue, and lastly did not understand what the user did not understand.

We need more ways to discover how people behave using computers, and then communicate that back to the designers and programmers. That is why I co-started Webnographer to help the designers and programmers understand how people use the software that they have created.

It boils down to what Stuart Card said “All designs are predictions, all predictions fail.”