Independent Evaluation of Q: The Importance of Understanding Impact
In health and social care, where quality improvement efforts often operate within complex and resource-limited systems, evaluations are vital for informing where time and resources should be invested in order to have the most impact on patients. Fundamentally, evaluations support transparency and facilitate evidence-based decision-making; guiding organisations to maintain or refine initiatives based on robust evidence. In this blog, we share insights from our evaluation of the Q community and reflect on the importance of understanding impact and generating actionable insights that support sustained improvement.
Evaluating the Q community
In 2023, Picker and research partners ResPeo and Healthcare Priority Solutions (HPS) were commissioned to evaluate the impact of the Q community. Q is a membership community of individuals working to improve the health and social care sector. The evaluation used a combined realist and performance approach to understand what activities that Q hosted were working, for whom, in what circumstances, and why, and subsequently assess the cost effectiveness of each activity. The research team used a mixed-method approach to data collection and analysis which included: interviews, workshops, case studies and surveys. Overall, nearly 700 people were involved in the evaluation – this included Q members, system leaders, and individuals who are not members.
Alongside the main evaluation, two further deep dive evaluations were conducted into Q’s flagship offers: Q Lab and Q Exchange. These evaluations had their own distinct methodologies, allowing for a more targeted exploration of which elements of these initiatives created impact. Findings from the deep dive evaluations can be found in the full report.
Impact at individual level
The strongest impact of the community was identified at individual-member level, with those who experienced impact reporting increased motivation to deliver improvement work, and gains in knowledge, skills and confidence. Individuals that engaged with activities that enabled them to connect, collaborate and learn from peers were more likely to think Q had a positive impact than those who did not. For individuals working in isolation or operating within organisations that lacked support or resources for improvement, they were able to find this in the Q community. For some, Q provided a space to break down information silos, fostering connections and opportunities to work with others in different organisations, which ultimately benefited their personal development.
Organisational and system-level impact
Lower levels of impact were identified at organisational and system level. Members reported a number of contextual factors hindering impact, such as a lack of organisational support for quality improvement and a lack of capacity to deliver improvement work, which can make it challenging for organisational impact to be felt. It should be acknowledged that identifying impact at these levels was more challenging to define, measure and analyse given the varying contexts of members and the complexity of interactions outside of Q. It is necessary to also consider the level of impact alongside reasonable expectations for a community that focuses primarily on engaging individual community members. That said, some of Q’s activities were shown to have ultimate impact – direct positive impact on patients – despite the indirect nature of Q’s work. Examples of this were best seen in the projects Q funded, which at times were rolled out directly to clinical staff and / or patients, and saw new approaches launched to better support care delivery.
Engagement as a driver of impact
The level of engagement with the community played a key role in an individual’s perception of impact, with those who engaged more with the community identifying higher levels of impact; suggesting the more you interact with the community, the more impact you will feel. Although, there were a number of reasons why engagement with Q was limited. Members reported limited engagement due to contextual factors such as poor system readiness, and Q community factors such as perceived relevance, inclusivity and accessibility.
From insight to action
The evaluation identified clear areas for improvement, where Q should focus attention in order to increase its impact for individuals, organisations and the wider health and care system moving forwards. Twenty-one recommendations were provided which focused on offer-specific recommendations, content recommendations (not specific to a single offer) and operational and strategic recommendations. These recommendations took into account Q’s strategy to date, and aimed to support Q to increase engagement and impact, as it works to strengthen and maintain organisational resilience and agility to achieve their vision.
Findings from the evaluation have already begun to influence decision-making within Q, shaping strategic direction and choices surrounding different activities. The Q team has welcomed the finding of the report and have articulated the steps they will, and have already begun, taking to address the recommendations.
“We used early insights from this evaluation to adapt to some significant changes to our funding context that we hadn’t anticipated when this project was commissioned. We are now applying what we’ve learned from the analysis, across all areas of our work.”
Penny Pereira, Q Managing Director
“The Picker-led evaluation has provided high quality data collection and analysis of Q’s impact to date. But most importantly, Picker’s strong commitment to collecting meaningful and actionable learning has meant the work is directly informing our strategic direction and choices. The evaluation has been conducted rigorously and independently but never at the expense of being inclusive of diverse perspectives across our community and being highly flexible in responding to our evolving needs. I wouldn’t hesitate to work with Picker again and would recommend them to any organisation looking for an evaluation partner that combines technical excellence with a collaborative approach, focused on practical application.”
Matthew Hill, Q Head of Evaluation and Insight
Looking ahead
In a health and care system facing sustained pressure and complexity, communities like Q play an important role in supporting those on the ground leading change. The evaluation revealed clear areas of impact arising from Q creating opportunities to learn, connect, share, and collaborate. With a clear picture in mind of how Q creates impact, and its barriers, the actions now being taken by the community will enable impact to increase moving forwards.
To read more about the Independent Evaluation of Q, please read the summary report.
If you would like to discuss this blog further or are interested in understanding more about our evaluation work, please contact us at [email protected].
