Survey length is one of the most important considerations in quality research – generally the shorter, the better, but you also have to balance this with meeting your objectives.
This is perhaps one of the most difficult challenges in doing research. While it’s generally the case that you want to get as much information as you can in one project, long surveys generally lead to poorer quality of response and subsequent data. Consider this – longer surveys have been shown consistently to lead to:
- respondent fatigue, which means lower levels of attention to your questions and therefore poorer data,
- more terminations amongst those who have been good enough to start your survey,
- a poorer experience with your survey and therefore the brand or organisation they represent, and
- lower response rates in the future, making subsequent research more difficult.
It’s important to remember that all research relies on the goodwill of respondents. Longer surveys generally lead to a diminishment of this goodwill and that’s not a good thing for potential respondents or researchers.
It’s also important to consider that a specified survey length is a bit of nonsense because it is based on an average, and we all own one-seventh of a Labrador on average! That is, in all surveys there will be respondents that finish faster or slower than others, which you need to consider. Try and define it in terms of a range, such as by using 5-minute blocks – e.g. less than 5 minutes, 6–10 minutes, 11–15 minutes, etc.
So, what’s the best guideline for survey length?
Try to keep your surveys to 5–15 minutes and offer incentives for longer surveys. A 5–10-minute survey is usually not an issue; most people will do these sorts of surveys if they are engaged with the subject matter or offered an appropriate incentive. Longer surveys can be conducted but you need to think hard about who you are researching and what’s in it for them.
The subject of incentives is a contentious issue in research and beyond the scope of this post. Generally there are those that think that incentives are “cash for comments” and create a bias. The alternate argument is that the lack of incentives also creates bias because it means some you miss out on certain segments of people – usually those who are more time poor and not willing to do a survey for nothing. The decision should always be made based on three considerations outlined below.
Length of survey
It’s a good idea to consider offering an incentive for surveys over 10 minutes and to offer an incentive for surveys over 15 minutes. For surveys over 20 minutes, you really need to consider an incentive that is commensurate with the time taken to complete.
Type of respondent
If you are talking to a general consumer, then you can probably offer an incentive such as a chance to win a pool prize or a token cash incentive that shows that you appreciate their time. If you are speaking to a senior person in an organisation or other time-poor respondents, then you really need to consider an incentive appropriate for their time.
Level of engagement/involvement
Low-involvement categories or subjects generally mean respondents will give you less time, while you might have a bit more room to move for high-involvement categories. In other words, people are willing to participate longer in subjects that they care about, like social issues, or where they spend longer times making decisions, such as for cars and holidays. It gives you a bit of room but don’t abuse it!
Optimising survey length
Firstly, optimise your questionnaire by ensuring that each question is matched to a specific objective. Think of it as each question earning the right to be there. If it’s a “nice to know” and you don’t have room, it’s out!
Although single-source data (all data collected from one respondent) is a wonderful thing from an analysis perspective, it isn’t necessarily so from the respondents’ perspective, who must sit through long and tedious questioning. If your objectives can’t be covered with a survey of suitable length, then consider running a split sample (two identical samples covering different things) or consider running two separate studies over time with objectives split between them.
If your survey is a bit longer than you are comfortable with, think about making the survey itself more engaging by using techniques like visually interactive question types using drag-and-drop, image choosers, image ratings, interactive videos, sliders and the like. Most survey software platforms now offer these, but you need to be careful as sometimes they too can add to the task and don’t always work on small-screen devices like mobiles.
Giving respondents the opportunity to save their progress is also a useful technique if you are worried about the survey length. Basically, you give respondents a “come back later” code which enables them to finish the survey in two or more sittings without losing any continuity in the data. You can send reminders too.
Finally, it’s a very good idea to tell respondents how long the survey will realistically take and display their progress throughout the survey via a progress bar. Dividing your survey into logical sections can also help, as can thanking and encouraging respondents throughout the survey journey.
Hopefully these guidelines, tips and tricks will help you create shorter, more engaging surveys. In our experience, questionnaires tend end up longer than they should be. Making sure you are aware of survey length and always doing what you can to reduce and optimise it will help create better responses and better quality data that your decisions are going to be made on.