Okay, so you’ve set out your decision-centric goals and objectives and done your initial planning. It’s now time to put pen to paper and start thinking about the research instrument – that is the questionnaire. But before you start writing questions, think about a design and structure for the instrument.
Good questionnaires have flow and take into consideration the respondent, the logical introduction of topics and specific areas of investigation. You also need to consider the impact of bias. Bias is relevant to your design and structure as the presence of some questions may impact later responses. (For more info on common sources of bias and strategies to minimise them, see our post “Minimising Bias“)
A great example is unaided brand awareness; it’s unwise to ask respondents which brands they’ve purchased from a list, then to ask them which brands they are aware of. Why? It might seem obvious, but you have just prompted them with a list of brands so your “top of mind” awareness responses will be biased.
So how do you create a design plan for a questionnaire? Think of a questionnaire like a book with chapters, with each chapter addressing a specific topic. Plan your chapters to roughly address your objectives, although you may cover some objectives in more than one chapter, or a single objective with different questions in different chapters. Either way, it’s good practice to consider and plan these out.
Before you start planning out your chapters, think about whom you are going to let read your book. The very first thing you therefore need to consider is who will qualify for your survey – that is, who is the target respondent? It’s pretty unusual that you want to speak to everyone, and at the same time it’s also pretty unusual to find a source of respondents that fits your target exactly. So you need to think about screening your respondents.
Screening is the process whereby you “screen in” (or let through) people who qualify for you survey and “screen out” (or terminate) people who don’t qualify. A simple example would be with gender, age and area. We might only be interested in speaking to females aged 25–54 who live in Sydney. We would need to ask these demographics questions right at the front of our survey to make sure we’ve got the right people. It’s a waste of resources and respondents to do this at the end because you will end up discarding those responses that don’t qualify.
Screening usually goes beyond simple demographics and typically includes some sort of behavioural or category purchase questions. A good example might be being the grocery buyer of the household and buying in a particular category like frozen vegetables, so you’ll need to ask these questions up front as well. It’s a really good idea to mask these types of questions to reduce self-selection bias, especially if you are rewarding respondents who qualify with incentives. Using the aforementioned example, the screener questions might be:
Which of these are you personally responsible for in the household you live in?
Doing the housework
Paying the bills
Buying the groceries
Maintaining a car
Which of these have you purchased for this household in the last 6 months?
Frozen pre-made meals
This way, a respondent doesn’t know the right answer to continue, and is therefore more likely to give truthful answers rather than answers that will get them into the survey and therefore access to the incentive. You would then only allow those who selected the third choice for both questions above to complete your survey, and terminate and thank the rest for their time.
So, now on to the “book” itself in our earlier analogy. The first thing to consider is an introduction. You wouldn’t write a book without it, so you need to start broadly. Introduce the topic and then drill down to specifics, allowing logical breaks and changes of topics that flow and fit a natural way of thinking.
A typical plan might look like the following, again using the previous example of frozen vegetables:
- Screening questions (key demographics and masked behaviour and/or category questions).
- Category introduction (broad). Thinking now about frozen vegetables… Ask broad questions such as frequency of purchase and use, who consumes them, where and when purchased etc. This gets your respondents thinking about the category generally.
- Category brands (specific). Which brands can you think of? Which of these have you heard of before today? Which have you ever bought/do you currently buy? Which is your main brand? etc. This gets your respondents to now think about the brands they know and use.
- Category drivers (specific). How important are each of these in which brand you choose? Now that they have thought about the category and the brands, they are better positioned to think about the factors and attributes that drive brand choice.
- Brand performance. How well do each of these brands deliver on these factors? With these factors and attributes in mind, they can assess brands on how well they deliver.
- Demographics. Questions like marital and work status, education level, household composition, etc.
It’s a really good idea to end your survey with demographics so that you can classify your responses and look for differences between key groups. It’s also a good place to ask any sensitive questions like income which might put people off if placed earlier in the survey, leading to terminations.
Once you have thought out your plan, you have a blueprint for where your questions are going to go and how it will flow. This might change as you start to write your questions, but is a good practice to make the whole process easier and better for your respondents’ experience and data quality. Finally, check your topics or “chapters” against your objectives to make sure you have a place for them to be covered.
You’re now ready to start drafting your questionnaire – good luck!