Are patient-reported experience measures of psychometric evaluation valid?
Picker’s clients and partners – quite naturally – want to know that they can measure service-user experience with ‘validated’ instruments. Picker takes validation seriously and gathers evidence to support the use of its questionnaires for evaluating patient care and for quality improvement.
As with our mission in general, we centre our validation work on service users themselves, drawing on their experience in focus groups and interviews to determine the scope and content of the questions, to test our interpretation of their answers to those questions and to ensure that people can respond in ways that properly convey their experience of care. We use statistical methods too, to detect where respondents may be finding it difficult to answer or where questions are not providing the level of information we need to evaluate care.
Psychometricians have, over many years, developed a suite of statistical methods to assess the validity and reliability of questionnaire data. Their origins lie in the fields of educational and psychological testing. These techniques cannot automatically be applied to the development of patient-reported experience measurement because the nature of what is being measured is quite different from psychological traits or abilities. However, this is not always recognised by researchers or by journal reviewers. In an article published in the International Journal for Quality in Health Care, Steve Sizmur and Chris Graham from Picker and Nanne Bos from the Netherlands Institute for Health Services Research, argue that statistical validation methods need to be matched to the measurement model appropriate for the instrument under development.
For the psychometric measurement model to be valid, there must be an underlying influence that affects how someone will respond to a set of questions, in the way that (for example) arithmetic ability underlies how you will respond to an arithmetic test. Your ability precedes you taking the test and it determines how many questions you will get right. In a healthcare setting, there are multiple different causes behind how someone experiences their care, including individual members of staff, the design of healthcare facilities and who provides the catering. Someone who has a positive experience of their hospital food is not necessarily more likely to be treated with respect by clinicians and may not have a quieter, more comfortable room.
It makes more sense to say that patient experience is an evaluative response to these diverse events rather than their cause. This means that the neat mathematical properties of the psychometric model – which depend on the way that responses are correlated – do not hold. Psychometric tools, therefore, need to be chosen with care. Applying the wrong statistical criteria in survey development can lead to distortions in the questionnaire, which might not then fully reflect service users’ priorities.
The Picker Principles of Person Centred Care provide a conceptual blueprint for measuring patient experience. When developing a questionnaire or instrument to evaluate person-centred care, we select the most appropriate validation techniques and strive to develop and continually improve methods that are suited to this end. This may well include psychometric techniques, but only those that are sympathetic to this conceptual model.
