The Babylonians, who were famous for their astrological observations and calculations (aided by their invention of the abacus)
Last night Martyn Perks gave a talk at the monthly UPA event in London arguing that creativity is stifled by an over reliance on metrics. He argued that insight for good design can not come from number crunching.
I spent allot of Martyn’s talk shaking my head.
One of the points that he made was that Usability was driven by metrics. I could not agree, because there are few people using real metrics to measure usability. Jeff Sauro and his SUM method is one of the few exceptions. The practise of the most common methods are dominated by Expert Reviews and Lab Studies with few participants. Many of the reports that I have seen are dominated by opinion and are not objective.
I can understand Martyn’s anger at some of methods that are used to collect and analyse numbers. As we have found with Webnographer it is easy to collect data, what is hard is to find out what the data means. It is easy to jump to conclusions from the data, that don’t hold out to be true.
Do google searches translate into polling victory? This chart shows Palin having more searches than the other candidates.
A good example of misleading data is people trying to make sense of Google’s trends. If you look at the chart above it may make you jump to the conclusion that Palin is leading the race to the Whitehouse. Obviously this is not the case. (Thanks otrops for the link.)
The argument that I made to Martyn is that metrics can help at the right stage, and make the difference between an implementation of a design that is good, and one that is great. Many great designs from the walkman to the iPod have been dominated by a great ideas person, Akio Morita or Steve Jobs for example, who pushes through the great leap in innovation. After the initial design there are many versions of the same product, each one an improvement on the last.
The research used to inform the changes to the product should be tested using Sientific Method to see if the numbers that one is using hold true.
While data gathering and metrics can not come up with an idea, they can help to refine a product and improve it.
