Once you have programmed your questionnaire, take the time to thoroughly test it including all the skip logic. Testing is probably the most critical part of any project because it’s where you ensure that everything you have planned and designed in the paper (or “soft”) version of your questionnaire is executed correctly online. Once you have programmed your questionnaire, take the time to thoroughly test it including all the skip logic.
Testing should really be viewed as part of the questionnaire development process as some things that seem to work okay on paper don’t necessarily work as well once programmed. This is your last chance to refine and make sure everything is right, so make sure you allocate adequate time to the process in the timeline. We recommend at least two days for this process.
There are several things that need to be undertaken when testing and timing; a good process is paramount. Here are the critical steps we use to ensure a thorough testing and timing process.
Online versions of your questionnaire are usually created by copying and pasting text from the soft version of the questionnaire, but that does not mean there won’t be mistakes. Before you do anything else, create a printed master questionnaire for testing that you will mark up as you go.
The first step is to go through each question in the online version and tick off the all the instruction and question wording and response code options on your printed questionnaire. At this stage, don’t worry about skip logic, just follow the questionnaire, making sure that you select options that enable you to see every question in the survey. Sometimes this will require going backwards and forwards for mutually exclusive questions.
It’s probably a very good idea to do a spell check on your original questionnaire before copying and pasting, so you don’t have any typos in your online survey. You might want to think about which words need to be bolded or highlighted in the online version to make sure that important words and instructions stand out; always consider the user experience.
You will then need to check all the routing and piping or skip logic. This is the most important part of testing because, once you go live, it’s difficult to fix.
Although it can be a laborious task, the best way to do this is by moving back and forth through each section and question and make sure that you end up at the right spot. If it is a section jump, then check that the right people are qualifying and all of those that don’t jump to the next section.
If you have multiple GO TO statements or jumps from a single question, test every combination by going back and forth. Mark off iterations to confirm that the jump or skip was right. Keep going until all skips and jumps have been tested and validated as correct.
You will also need to check that all your piping is working in the question text (INSERT) and that relevant codes are displayed (SHOW). For example, if you have a question moving from:
Q1. Which of these brands have you ever heard of?
Q2. And which of these brands have you ever tried?
SHOW CODES AWARE OF IN Q1
…then select one brand at a time in the source question (Q1 in this case) and make sure all the others do not show in the next question.
UX piloting and timing
Next, it’s time to get someone else to pilot your survey, preferably someone who hasn’t seen it but would qualify for the survey and therefore know what you are asking. Usually one of your colleagues can do this for you. Ask for their candid feedback about the whole survey and if there was anything they didn’t understand or wasn’t clear. You might also ask at this time whether they enjoyed it or not, and if they felt that it was too long or not. It is best that you get a few people to do this as they will experience and pick up on different things.
You should also ask your pilot person to run a stopwatch and time how long it takes to complete the survey, asking them to read all instructions and filling it in as if they were answering it for real. You can time it yourself but we find that survey authors always fill in their own surveys faster as they are familiar with the structure and questions. It’s best to get a realistic timing as this will potentially impact your cost, especially if you are paying a panel to find respondents for you.
Ideally your online survey will be device-agnostic or have a “responsive” design, meaning it will render well on a variety of devices and window or screen sizes. It’s a good idea to make sure that the survey software you are using can detect the type of device that is being used and then is able to optimise the survey accordingly.
As an example, grids of statements using agree/disagree scales are usually terrible to complete on a smartphone and should automatically be converted to single questions. The question text would also need to wrap for the smaller screen size.
Other things to check for are the use of images, especially stimulus with writing embedded in the picture such as in copy testing of an ad. Can it be read? Or do you need to rescale the image, or display tooltips, or provide other options to display larger text?
Finally, you need to see what the load times are for images and videos and make sure they are acceptable in terms of time and size. People will not want to download large videos on their mobile devices, so spend time making sure you have the best compromise of size and quality.
Usually once you have completed all of your testing, it’s time to run a soft launch. We recommend that you only launch your survey to 10% of your sample so you can ensure that everything works well and all the questions are being answered correctly. You should also have a help link with an email and check that no-one has emailed you with queries or issues with the survey.
When you finally launch your survey, check the data after the first day to ensure that all the right questions are been appropriately completed. The best way to do this is to run a hole count with natural filters applied (i.e. filter each question according to the routing) to ensure that everyone who is supposed to be asked a question actually is.
Once you have passed through all these stages, you are good for launching to the rest of the sample.