Cognitive interviewing is an important stage in developing a questionnaire. The purpose is to find out how people interpret and answer questions so that we can understand if the questions are measuring what is intended.
The approach to cognitive testing involves the following steps:
In the past we have most often carried out interviews face-to-face, which gives researchers access to visual cues and clues that can indicate when participants may be struggling with a question. However, due to COVID we are increasingly conducting interviews via video calls (e.g. Skype, Zoom) which we have found to work well. They have proved to be cost-effective and allow for a wider geographical distribution of participants. A recent study found online technologies can be used to achieve a diverse sample of participants, and they called for other researchers to use such methods to ensure that hard-to-reach populations are represented in research .
Cognitive interviewing is important in ensuring survey questions are understood in the way they are intended. When a researcher finds that the participant is thinking about something different when they answer a particular question, there may be a problem with how the question is asked and it may need some revision.
We also want to know whether participants can retrieve the information required to answer a question from their memory. Generally, our recall period would be within a six to twelve-month period. However, this timeframe would be reconsidered if participants have difficulty recalling information accurately.
Many of our survey questions at Picker are closed questions, meaning they have a set number of response options to choose from. Cognitive interviewing also allows us to identify whether there are sufficient response options provided and whether the wording is suitable.
We would always set out to test the survey in the same format it will be used for data collection – whether it be paper, online, telephone, or a mixed methodology.
Cognitive interviewing ensures that the survey consists of a set of questions that are robust and fit-for-purpose and enable respondents to provide appropriate information. Some examples of how questions have been improved during this process are shared below.
Example 1: question-wording change
When carrying out cognitive interviews in the development of a hypothyroidism patient survey, the question ‘When were you diagnosed with hypothyroidism?’ was changed to ‘When did you develop hypothyroidism?’ This change accommodated participants who had not been officially diagnosed with hypothyroidism but who had self-diagnosed according to their symptoms.
Example 2: response option amendment
When cognitively testing a paediatric diabetes survey, the statement ‘I feel that our GP has a good understanding of diabetes’ was presented, with the following response options:
During the interviews it was apparent that many parents wouldn’t go to their GP for diabetes advice or support. As it stood, there wasn’t an option to accurately represent potentially many respondents. Therefore, the option ‘we don’t usually speak to our GP about diabetes’ was added.
Whilst these changes may seem small, they can have an important impact on data quality. Respondents who do not understand a question or who cannot find a suitable answer may ignore it or provide an unusable response (eg by selecting multiple options or writing in a comment of their own). Enabling people to respond appropriately maximises the volume of data available, supporting effective analysis.
Cognitive interviewing is a key stage in the development of survey questions. By focusing on the participant’s thought processes when answering a question, we can identify if there are any problems with the question wording or responses, and make any necessary changes to ensure the questions work as intended. This is important to ensure that the survey questions are appropriate and provide accurate and meaningful data.
 Upadhyay, U.D., Lipkovich, H. Using online technologies to improve diversity and inclusion in cognitive interviews with young people. BMC Med Res Methodol 20, 159 (2020). https://doi.org/10.1186/s12874-020-01024-9
items marked with * are required