blog

What Does It All Mean?

Thu, 17 May 2012 15:32:01 +0100

As I crawled around the M25 on my way home last week, several news items on the radio caused me to question the way statistics are interpreted and reported in the public domain.

Confusing interpretation of numbers

Visiting a factory in Essex to underline the government’s commitment to austerity measures, an employee asked David Cameron why Conservatives and Liberal Democrats could not agree on new policies. The Prime Minister replied that forming new policies was inevitably more difficult because “you, the voters, decided that no one won the last election”. It is a puzzling (perhaps convenient) interpretation and not the first time politicians have suggested that the British public voted for a coalition government. It reflects the outcome more than the wish of individual voters. To use a football analogy, it’s like saying that a Wembley FA Cup Final crowd can’t decide which team should lift the trophy and so they settle for a draw. The truth, of course, is that the crowd’s support is clearly divided between the two teams. The same could be said of TV shows like The X Factor. The vote for two performers may well be very close, but that shouldn’t be mistaken for an indecisive audience. Every viewer has a preferred winner and this is reflected in how they cast their vote.

The second story related to the disturbing case of abuse in Oldham. “Nine men jailed for a total of 71 years” said the headline. What, exactly, does this mean? And why is the number 71 relevant in this story? In and of itself, it seems meaningless. Or does the magnitude of the total sentence convey the severity of the crime? Does it help the listener judge whether the sentence for any of the nine men is appropriate? Would it make the listener draw comparisons with similar cases in which multiple prison sentences had been handed out? I don’t think so. Surely the more relevant message is that each man was jailed for between 6 and 11 years?

On Twitter’s 5^th birthday, the company issued a press release boasting that “Twitter users send 1 billion tweets per week”. An impressively large number that captures the attention you might argue, but does it convey meaning? Only with further interrogation does the reader discover how this translates into the number of tweets per user per day, a much more relevant currency.

Implications for Market Research

Important parallels exist between these stories and our work. The MR equivalent of the coalition story would lead us to examine the distribution of ratings, not just the average. A new concept might be loved by some people and hated by others. It would be misleading to convey the average as the outcome. Likewise in the Oldham case, we would be unlikely to say that 9 concepts rated on a 1-10 scale attracted an overall score of 71. To promote wider adoption and appropriate use of MR requires researchers to interpret data correctly and present it in a clear and meaningful way.

And, on that note, I’m off to watch the football. And I won’t be satisfied with a draw...

The tyranny of choice

Thu, 10 May 2012 10:33:13 +0100

Recently a client came to us with a range of 58 different concepts for evaluation. This presented many tricky issues for the design of the research, and lead to a lively discussion with my colleagues regarding how many concepts a respondent could evaluate meaningfully in one survey, and how this compared to the decision making process in real life purchase situations. This in turn got me thinking about a couple of articles I had read recently about how increasing choice for consumers is creating an interesting effect. First was an article from Harvard Business Review. Its authors argued that the notion of the purchase funnel has in some instances been sidelined by other effects, most notably the purchase ‘tunnel’. This “tunnel” describes how consumers avoid the cognitive effort of careful consideration, and instead make simple choices based on the “easiest option”. Second was an old article I had read in The Economist, which talked about the ‘tyranny of choice’.

The tyranny of choice

As we have become collectively “better off”, and as distribution channels have become increasingly efficient, more and more products have become available to us. In many situations we are simply overwhelmed by choice. In the seventies the average supermarket carried about 10,000 lines. Nowadays that figure is closer to 50,000. Consider orange juice. There was a time when you had a choice of two or three different types of orange juice at the supermarket. Nowadays it’s not unusual to be faced with options that run well into double figures.

The received wisdom would have us believe that increasing choice will make us happier: after all, amongst all the options available to us we should be able to find the one that really ticks all our boxes. In fact, the reverse can be true. While an increasing range of product variants and brands has accompanied improvements in quality of life in the Western world, in actual fact there comes a point where increased choice becomes stressful.

To experience this for yourself, one simply has to walk into the local purveyor of mobile phones and be bamboozled by the choices on offer. Handsets, price plans, insurance options. I just want a phone! The information I have to take in is too much and I leave befuddled – to go away and do some more research online. That's why the insurance comparison sites have done so well. They curate the mass of choices down to a manageable number for you to compare based on your basic needs.

Implications for research

Thinking back to our initial conundrum of how to evaluate 58 different concept options, many closely related, we can see that the tyranny of choice also has implications for survey design. How does one go about replicating the real-world tyranny of choice in survey design? How do we avoid choice related studies from being too artificial, or inducing mind-numbing respondent fatigue (thereby producing poor quality data)? If consumers are now becoming used to choosing randomly (as suggested by the Harvard Business Review article), without research or using a more open-ended approach what does this mean for survey design?

It’s important that we consider these consumer purchasing and research effects in our work. In relation to the 58 concepts, we worked with our client on a two stage research design, which allowed us to explore a reduced number of concepts in a bit more depth. We also made sure that the concepts were reasonably different from each other. (Asking respondents to distinguish between ‘shades of grey’ is a sure-fire way to upset them and end up with poor quality data).

I was reading about the new Rolls-Royce Phantom last week. Apparently it has an eight speed gear box, but you can’t choose which one you are in at any time. You can go forwards, backwards, or stay where you are. Could it be that having choice removed completely is becoming the ultimate luxury?

Do you know who is analysing your data?

Thu, 03 May 2012 17:23:31 +0100

We are often asked, just what are “Advanced Analytics”? In response to this, I usually begin by saying that what they’re not are web-based techniques that measure social media traffic etc. This has become almost an industry within itself over the last few years, and confusingly is referred to generally as “Analytics”. So it’s important to make this distinction.

Getting curious – exploring hidden relationships

Advanced Analytics are analytical techniques that go beyond just describing the data: instead, they explore the hidden relationships and patterns within data. They allow us to infer things that are going on that can’t be seen with more simple cross-tab analysis. (That’s why, in statistical circles, advanced analytic techniques are often referred to as inferential statistics). And this is one of the things I really enjoy about my job: finding things in the research and data that aren’t immediately apparent, and that other people don’t necessarily see.

There are many techniques that come under the general banner of advanced analytics. Perhaps the most common one people have heard of is correlation analysis, typically used in key drivers analysis. Others include conjoint, maxdiff, factor and principal components analysis, structural equation modelling, cluster analysis, latent class modelling, linear and non-linear regression, Kruskal’s relative importance analysis, chi-square – the list of techniques available to the advanced analyst is extensive. For me, that’s one of the best things about being an analyst – you never run out of things to learn. Trusted techniques such as conjoint analysis are continually evolving, and new ones are being developed. Every year, new software versions are released which offer significant improvements on previous techniques. Keeping up with (and hopefully sometimes leading) these developments is a challenge, but a very interesting and worthwhile one.

Getting creative with the numbers

What doesn’t fit with most people’s perceptions of what advanced analytics is about is that it’s very creative. As the lead analyst on a research project, the choices you make determine the results more directly than any other team member. Whether it’s deciding on a specific data transformation to create a range of segment solutions, or how to group potential drivers in a customer satisfaction analysis, these decisions ultimately rest with the analyst and their knowledge of the business objectives of the analysis.

A lot of companies only have one or two analysts, and in such a context it’s easy to become pigeon-holed as a number crunching boffin. At Bonamy Finch we have a big team of commercially astute analysts who can bounce ideas off one another. We like to work in a way where the analyst has a full understanding of the wider project, its context and business objectives. We don’t just crunch the numbers, we think about an intelligent end solution.

When selecting an agency how much thought do you give to who will be analysing your data?

A short note on the practical significance of being tall

Thu, 19 Apr 2012 15:07:34 +0100

I’m five feet nine inches tall. How would you describe me? Average height? A bit below average? Short? I’m guessing you wouldn’t call me tall. I can’t say that it was a conscious recruitment decision criterion, but amongst my colleagues this does actually make me above average. Only one of my colleagues is clearly taller than me (if you exclude our IT support chap, who at about 6 feet 7 inches is clearly an outlier).

When I was a teenager I learnt the difficulties of analytical forecasting the hard way by plotting my own growth chart. Unaware as I was at the time of ceiling and threshold effects, and by ignoring the fact that no member of my family had exceeded five feet eight inches, I predicted for myself a final achieved height (at the age of eighteen) of 6 feet. All went well for a couple of years, with model validity proving high, until at the age of sixteen things started to slow down. And then at some point, having achieved the not too giddying height of five feet nine inches, I stopped growing. Thankfully I had my sense of humour to compensate for lack of imposing physical stature. I also found solace in the fact that at the time the average male height in the UK was exactly five feet nine inches. So I was, in this respect, reassuringly average. And definitely not short.

US TV Advert Shocker

I was in the US last week, and an advert came on the television for shoe inserts that promised an instant height increase of three inches. The advert showed a man standing next to an attractive woman who he clearly fancied and who equally clearly was not the slightest bit interested in him. A quick visit to the “Magi-Lift insoles” website, and a three inch lift later, the same man walks up to the same woman and she is all over him like a bad suit. Watching US TV adverts can quickly immure you to the asinine, but what particularly seized my attention was the man’s “before” height: five feet nine inches. Any illusory beliefs I had clung to in my adult life of average heightiness were cruelly, and irreparably, shattered.

Being short is no small matter

Whatever. As I say I have my sense of humour to fall back on. But it does bear upon an issue that I initially raised in my blog a couple of weeks ago: that of the practical (as opposed to statistical) significance of differences we might report in our research. At five feet nine inches I am, (by US TV advert standards at least), short, but if I had told you I was six feet in my opening sentence, you might well have described me as tall. Yet the latter is only 4% greater than the former. So we have a small difference in absolute terms, but one of very practical significance in terms of how I am perceived by others. For instance, tall people are perceived as more confident, successful and attractive than short people, are more likely to be given a job interview, get more responses from personal adverts, and are also more likely to become President of the United States*.

What does this mean to researchers?

Now consider a research debrief in which one new product concept has 4% higher appeal than another. Or Brand A has 4% higher awareness than Brand B. Or the Southern region has after-sales service satisfaction 4% higher than the Northern region. Whether or not these differences are statistically significant will depend largely on how many respondents the research budget could stretch to, and the variation in peoples’ responses. But are they practically significant? Will the first concept perform better in the marketplace? Does Brand A convert more people to purchase than Brand B? Should the Northern region follow the practices of the Southern region? Who’s to say?

Well, we are. Professional researchers need to understand the limitations of statistical significance, and how to draw conclusions about the practical significance of differences and patterns in our data. If we can’t do this effectively, then we are not in a position to make well founded business recommendations to our clients. My feeling is that it’s not something our industry excels at, but I could of course be wrong!

Although that might not be significant.

* I remember the following factoid from one of my social psychology textbooks: the vast majority of US presidents have been taller than the defeated candidate, and in the few instances where they have been shorter, they are generally actually perceived by voters as being taller.

Money talks: Rewarding better survey design

Thu, 12 Apr 2012 17:35:59 +0100

Long. Boring. Poorly written. Irrelevant. Repetitive. Just some of the widely documented criticisms levelled at many of today’s market research surveys. Much of what we read and hear anecdotally suggests our profession continues to create a negative experience for survey takers. They endure dull questionnaires and are left feeling frustrated, with a negative impression of market research. Why does this situation prevail?

Where’s the motivation to improve?

It strikes me that researchers have no real motive to address this issue. There’s an almost endless supply of panellists out there and it doesn’t matter if we upset a few of them, does it? As long as we get our data, we’re happy. Anyway, the panellists are getting paid for their time. But it’s ironic that, in designing the tools to “listen to the customer”, the MR industry fails to do exactly that. Is anyone really thinking about the poor respondent? What about the data quality?

This made me think about how other industries operate in order to create a competitive market place and weed out the weakest performers. These days we are invited to rate just about anything and everything – the latest book or CD bought on Amazon, your recent stay at a hotel, the service you received at your bank. The list goes on. And people take notice of ratings.

Systematic rating of surveys

Could the same model be applied to data collection in the online world? Why not adopt an Amazon style rating system for surveys and apply it to data collection across the industry? If MR really cares about respondents and we all want top quality data from our research, why not ask the very people completing those surveys what they think? Then reward or penalise the MR agency according to what the people say.

How would this work in practice? Across all panel companies, respondents would be asked to rate every survey they take on a standard “enjoyment” (or similar) scale. Authors of the most enjoyable surveys (as rated by respondents) would be rewarded with favourable data collection cost rates (because fewer drop-outs, more enjoyment / satisfaction, less churn etc.). Authors of the worst surveys would be penalised with higher cost rates. This is no different to, say, the car insurance market – those displaying the best behaviour are effectively subsidised by those deemed most risky / likely to offend.

Is this the future?

This would surely provide a tangible motive for every survey writer to brush up his/her survey design skills. It would create a currency (think NPS) through which MR agencies could sell themselves to research buyers. Which agency wouldn’t covet the title “best survey writer 2013”? Data quality would inevitably improve. Competition would intensify and standards would be driven up. And let’s not forget the ever suffering respondent, who would encounter more innovative, thoughtful and shorter surveys.

Could an industry-wide solution work in practice? Survey writers – how would you feel about your ratings being publically available? Research buyers – would this solution influence who you choose to work with?

Red Meat, 'Wonder Drugs' and Flossing

Fri, 30 Mar 2012 11:42:27 +0100

A real brouhaha followed the recent wide reporting of research into the perilous effects of red meat consumption. Having been a veggie for several years in my youth my interest was well and truly piqued. The headline finding was that “eating an extra portion of red meat every day will increase your risk of death by 13% annually”. Given that, with the notable exception of earthbound deities, our risk of death is nominally considered to be 100%, the headline by itself raises some interesting issues about our use and interpretation of statistics.

But what does this actually mean?

No matter. What was particularly interesting in the reporting of this finding was what it actually means to people. Enter the dubiously titled Professor Spiegelhalter, a Cambridge University biostatistician (no, I didn’t know there were such things either) who is currently the Winton Professor of the Public Understanding of Risk (ditto). He said to imagine two versions of yourself at 40, one who eats just 3 ounces (85g) of red meat a day, and one who eats 6 ounces (170g). The second version of you would die at 79, the former at 80. Cue a big sigh of relief from the committed carnivores amongst us, for whom this difference seems rather small, and not nearly as important as a 13% increase in risk of death! But before breathing a collective sigh of relief and flossing the bits of meat from between their teeth, Prof Spiegelhalter suggests the committed meat eaters could instead view it as each time they are eat their extra daily hamburger, they are actually shortening their lives by more than half an hour! Suddenly it seems a bit more serious...

The way we frame data is all important

This is a very clear demonstration of the importance of how we, in the research and analytics business, not just analyse data, but how we frame it. We have probably all been in meetings discussing not just the statistical but practical significance of a 0.4 mean increase in customer satisfaction on a 10 point scale. What does it actually mean to the client? Is it enough to poach customers, drive loyalty, or differentiate their offering? The illustrations put forward by Prof. Spiegelhalter remind us of the power of framing data in a meaningful way.

Another debate surfaced again last week – that of the potential benefits versus risks associated with aspirin. Advocates portray it as a ‘wonder drug’, with continued low dosage drastically reducing risk of heart disease and cancer. Others, however, point to increased risks of internal bleeding as outweighing these potential benefits. During a considered, extensive (and very dry) debate on the latest aspirin research to be published in The Lancet, the research’s author repeatedly demurred as to whether people should take a daily aspirin. He suggested instead they ask their doctor. Finally the interviewer said “The people listening to this interview will want to know what you do. Do you take a daily aspirin?” To which he replied simply, “Yes”. And then there was closure – a clear signal on what the data was telling us, and what it actually means for people. The data and subtle points of argument were distilled into a simple question with a decisive answer. Again, there are clear parallels with how we (in the research and analytics business) should approach our own communication of research findings.

Keep flossing…

So, taken together with last week’s advice from the Dalai Lama, for a healthy and happy life, it seems that we should be compassionate to others, only eat a little red meat, and take a daily low dose aspirin (if we are over 40 and in a high risk group for heart disease or cancer). Oh, and something that hasn’t been so widely reported – floss every day. It’ll add years to your life. Really.

The Dalai Lama Does Key Drivers Analysis

Thu, 22 Mar 2012 17:31:44 +0000

I follow the Dalai Lama on Twitter (@DalaiLama), and he tweets regularly about the value of compassion in people’s lives. In fact I’d say that most of his postings address this theme in one way or another. For the Dalai Lama, compassion is the fundamental key to a happier life: a life spent acting out of concern for the welfare and happiness of others is itself a happy one. It’s a simple message, and, judging by the number of Buddhists in the world, a compelling one. Yesterday he didn’t mention compassion, but instead tweeted the following:

“Wealth may contribute to our happiness, but it’s not the most important factor; by itself wealth fails to bring us deep inner satisfaction”.

Thinking about Key Drivers Analysis

And this got me thinking about Key Drivers Analysis (KDA). Although he probably wouldn’t think of it in such terms, the Dalai Lama was offering his own model of the drivers of life satisfaction (happiness) based on a subjective KDA. Within his model, compassion might be regarded as the most important driver of happiness, and a necessary condition to fulfil in order to be truly happy. Wealth might be considered a secondary driver of happiness, but not a sufficient condition by itself to make one truly happy. You might also interpret his latest tweet as suggesting some kind of interaction effect, whereby wealth might interact with compassion to raise levels of happiness. I’d have to chat with him to see if that is what he is suggesting.

So, a few simple statements help to reveal the Dalai Lama’s implicitly held model of the key drivers of happiness. He has spent his whole life determining this model, and putting it into practice through his actions and thoughts. He clearly takes it seriously.

Problems with KDA

On a less spiritual level, at Bonamy Finch we also take our key drivers analyses seriously. Whether it’s working out what’s important, necessary or sufficient to make a customer come back for more, or say good things about you, or whether it’s exploring what brand personality attributes really push people’s buttons, we are always looking for better ways to arrive at the answers. For instance, the trusty correlation technique doesn’t help get past drivers that are highly related to each other. Regression often relies on complete datasets (where every respondent answers about every potential driver), and these are not too common. It can also throw out some very squiffy results when the drivers are related to each other. Structural equation modelling is perhaps the gold standard, but the data requirements needed to produce reliable results are often not met by research that is not designed with this analysis specifically in mind.

A better type of KDA

We have spent considerable effort exploring techniques that overcome these problems, and through this work have found very positive results using a statistical technique called Kruskal’s Relative Importance analysis. It’s not that common – we’ve had to write our own software to run it – but it produces results that are reliable and differentiate clearly between the importance of drivers. Not only that, but it can deal with missing data and variables that are strongly related to each other. For those reasons we have made it our default approach to KDA. (Although of course, we always look at the research thoroughly before deciding what the most appropriate approach is). We have written a concise overview of Kruskal’s Relative Importance for KDA, and how this can help deliver clearer, more reliable research results. Please contact us if you would like a pdf copy.

Now I have to go and start being compassionate…

A big welcome to Lorenza!

Mon, 13 Feb 2012 09:58:56 +0000

We are delighted to welcome Lorenza Edge to Bonamy Finch. As a Director in the Research Solutions team, Lorenza will be responsible for delivering full service research to our client base of strategic innovation agencies, brand consultants and other marketing services companies.

Her proven experience in providing strategic insight and consultancy makes her a strong addition to the team. Welcome Lorenza!

We are also pleased to announce the return of Corinne Macaskill after a year on maternity leave, following the birth of her second child. Welcome back Corinne!

Visit our people section to find out more about both Lorenza and Corinne.

Data Quality in Segmentations

Fri, 02 Sep 2011 17:45:03 +0100

I have just returned from two separate client meetings in Stockholm where we presented segmentations and the issue of data quality came up in both.

This issue seems to be particularly relevant to segmentations, as bad quality data is often conspicuously manifested in the existence of what is sometimes ignominiously known as a “dump cluster”. This will be a group of people who seem to want everything, or nothing, that a category has to offer. In reality they might actually be a group of people who didn’t engage with the survey, and didn’t give what we would regard as “thoughtful” answers. There are ways to try to discern if a “flat” segment actually represents a genuine facet of the market, but ideally analysts would not be faced with this task.

We’re working with our fieldwork suppliers to help reduce this problem at the source, and plan to publish our findings and guidelines later this year. What do you do about the dump cluster?

A New Era for Bonamy Finch

Wed, 06 Jul 2011 15:09:26 +0100

It seems like a very long time ago when I sat down at the desk in my loft and began my first Bonamy Finch project. And indeed it was. Six years and 5 days to be precise. A lot has happened in that time, and Bonamy Finch has developed in ways I couldn’t anticipate or imagine. I left the confines of my loft after two frantic and stuffy years, and we have been based in Coveham House for four years now. This is only five minutes walk from my house, but given that I became used to a 5 yard commute to my loft, it still seems like quite a schlep.

Market Research's Best Kept Secret

Going into our seventh year, we are at a very exciting time in the company’s development. We have strengthened both our Research Solutions and Advanced Analytics teams over the last year, grown by over 50% in the last two years, and developed strategic partnerships with an exciting range of new clients. A while ago one of our clients referred to us as one of the market research industry’s best kept secrets, and our aim now is to stop hiding our light under a bushel!

New Website and Increased Visibility

Now we have revamped our website. It clearly articulates how we deliver advanced analytics and research solutions in line with our brand values of Thought, Communication and Service. We will also be increasing our visibility at industry events, and expanding our product offering. Watch this space...