7.7. Testing survey questions: importance of pilot studies

It is always sensible to test a new or modified questionnaire in a small scale pilot survey before circulating it more widely to a larger group of survey participants. Inevitably in the answering and reviewing of the pilot questionnaires, some unanticipated problems will be highlighted, from minor issues such as duplication of question numbers, to misinterpretation of question wording and issues requiring modification of question wording, new response options and/or additional questions.

Box 8 gives an example.

Box 8. Case study: Pilot surveys in Scotland.

In recent surveys in Scotland, for example, about 6 people known to one of the survey organisers through his local beekeeping association were identified as suitable candidates who were readily contactable, covering a wide span of years of beekeeping experience from the beginner to the much more experienced. The questionnaire was delivered to them personally at a time when they able to deal with it immediately or an arrangement made to collect it shortly thereafter, so that no responses went missing. In the face to face situation, any immediate difficulties in understanding the questions are easily dealt with and explained, and a note made that these questions need to be re-worded. In all cases this exercise has suggested some points to be changed in the survey questionnaire, if only minor ones, and has been felt to be very useful.

If re-using a well-tried and tested questionnaire, clearly there is less need for a pilot run. However, if new questions are added a small pilot run is still advisable. Almost always some small point has been overlooked or can be improved upon, despite the most careful survey design. Even with an old questionnaire, piloting is often advisable to ensure that questions are still comprehensible and relevant.