What is cognitive interviewing?
Cognitive interviewing is an important stage in developing a questionnaire. The purpose is to find out how people interpret and answer questions so that we can understand if the questions are measuring what is intended.
The approach to cognitive testing involves the following steps:
- Recruitment: We recruit people who would be eligible to complete the survey – normally people who have used a particular service or had a particular health experience. We typically use a screening process to ensure we speak to people from a range of backgrounds (generally varying in age, gender, geographic region and ethnic background).
- Cognitive interview: Cognitive interviews involve a researcher observing a participant as they complete a questionnaire. During the interview, a researcher goes through the survey with a participant and asks them to ‘think aloud’ as they answer the questions. The researcher asks additional questions such as why the participant chose the response they did. This enables the researcher to identify where questions may not be as clear as they could be.
- Making changes and further testing: Any problems with the questionnaire that are made apparent during the interviews are noted. Interviews are carried out in rounds of usually 4-8 interviews at a time, after which any issues with questionnaire format, question wording or response options are reviewed by the researchers, to agree on what changes are to be made. The changes are then tested in the next round of interviews, and the process continues until there are no further issues arising.
In the past we have most often carried out interviews face-to-face, which gives researchers access to visual cues and clues that can indicate when participants may be struggling with a question. However, due to COVID we are increasingly conducting interviews via video calls (e.g. Skype, Zoom) which we have found to work well. They have proved to be cost-effective and allow for a wider geographical distribution of participants. A recent study found online technologies can be used to achieve a diverse sample of participants, and they called for other researchers to use such methods to ensure that hard-to-reach populations are represented in research [1].
Why is cognitive interviewing important?
Cognitive interviewing is important in ensuring survey questions are understood in the way they are intended. When a researcher finds that the participant is thinking about something different when they answer a particular question, there may be a problem with how the question is asked and it may need some revision.
We also want to know whether participants can retrieve the information required to answer a question from their memory. Generally, our recall period would be within a six to twelve-month period. However, this timeframe would be reconsidered if participants have difficulty recalling information accurately.
Many of our survey questions at Picker are closed questions, meaning they have a set number of response options to choose from. Cognitive interviewing also allows us to identify whether there are sufficient response options provided and whether the wording is suitable.
We would always set out to test the survey in the same format it will be used for data collection – whether it be paper, online, telephone, or a mixed methodology.
What does it achieve?
Cognitive interviewing ensures that the survey consists of a set of questions that are robust and fit-for-purpose and enable respondents to provide appropriate information. Some examples of how questions have been improved during this process are shared below.
Example 1: question-wording change
When carrying out cognitive interviews in the development of a hypothyroidism patient survey, the question ‘When were you diagnosed with hypothyroidism?’ was changed to ‘When did you develop hypothyroidism?’ This change accommodated participants who had not been officially diagnosed with hypothyroidism but who had self-diagnosed according to their symptoms.
Example 2: response option amendment
When cognitively testing a paediatric diabetes survey, the statement ‘I feel that our GP has a good understanding of diabetes’ was presented, with the following response options:
- Yes, definitely
- Yes, to some extent
- No
- Don’t know
During the interviews it was apparent that many parents wouldn’t go to their GP for diabetes advice or support. As it stood, there wasn’t an option to accurately represent potentially many respondents. Therefore, the option ‘we don’t usually speak to our GP about diabetes’ was added.
Whilst these changes may seem small, they can have an important impact on data quality. Respondents who do not understand a question or who cannot find a suitable answer may ignore it or provide an unusable response (eg by selecting multiple options or writing in a comment of their own). Enabling people to respond appropriately maximises the volume of data available, supporting effective analysis.
Summary
Cognitive interviewing is a key stage in the development of survey questions. By focusing on the participant’s thought processes when answering a question, we can identify if there are any problems with the question wording or responses, and make any necessary changes to ensure the questions work as intended. This is important to ensure that the survey questions are appropriate and provide accurate and meaningful data.
[1] Upadhyay, U.D., Lipkovich, H. Using online technologies to improve diversity and inclusion in cognitive interviews with young people. BMC Med Res Methodol 20, 159 (2020). https://doi.org/10.1186/s12874-020-01024-9
