Citizen participation in the set up of a performance measurement framework in Ireland
In a case study currently under review by Research Square and submitted in July 2021, researchers from the University of Amsterdam analysed the Irish initiative to involve citizens in the set-up of a performance assessment framework following a health system reform. The study, titled “Engaging citizens in the co-production of a health system performance assessment framework: a case study in Ireland”, focused on the method applied and the added value of using a citizen panel in the co-production of a health system performance assessment (HSPA) framework.
What is Health system performance assessment?
Health system performance measurement is proven to be useful in guiding health system actors to achieve improved outcomes and care experiences. Publicly reporting performance can also contribute to improvements, as well as transparency and accountability, which translates to increased public trust and a sense of ownership of the health care system. Different approaches to measure performance exist. Health System Performance Assessment (HSPA) is a country-specific participatory process, well-established and widely-supported mean to evaluate health systems as a whole. It is based on a limited number of indicators linking outcomes with system functions or strategies.
Context
In 2017, Ireland launched its Sláintecare 10-year reform program for healthcare. Important components of the reform were the development of a HSPA framework, but also the improvement of the governance, accountability, and performance of the health system. In 2019, the Irish Department of Health (DoH) and Health Service Executive (HSE) commissioned researchers to produce an outcome-oriented HSPA framework, which should reflect the shared priorities of multiple stakeholders, including citizens.
The study
Engagement of citizens in participatory policy processes links to the broader discussion on value-creating learning health systems. Being considerate of the central role of citizens’ care experiences and perceptions about the health care system can strengthen value creation and nurture public trust and a sense of ownership, contributing to a more patient-centred health care system that prioritizes care and outcomes that matter to patients. While methods for engaging stakeholders are well-established in their own right, citizen participation applied to HSPA processes has not yet been embedded into development cycles.
The HSPA framework was constructed with the inputs from three panels: one with health services professionals, one with professional associations, academia and patient organizations, and one with citizens. This study focuses on the citizen panel. Acting as representation of the Irish public, it included 15 people of varying walks of life. This study researches the use of methods for the engagement of citizens and other stakeholders for the development of Ireland’s first HSPA framework. It aims to describe the method applied in the Irish context and to reflect on the added value of using a citizen panel in the co-production of an HSPA framework.
Workshop
During the citizen panel workshop, their preferences on what information should be measured and reported on Ireland’s health care system was collected through a prioritization exercise. Then, citizens were tasked with identifying priority approaches and channels whereby information on the health care system could be reported back to the public. Specifically, the panel moderators presented the most commonly used themes to report on the performance of health care systems, as established by a literature review: health and well-being, equity, accessibility, quality of care, social and financial risk protection, coverage, safety, responsiveness, efficiency, effectiveness, people-centred, continuity of care, and coordination of care.
After each presentation, participants wrote down on sticky notes what measures or topics were of concern to them. A 3-colors scheme was used to measure the degree of importance assigned to a measure/topic, from extremely important to somewhat important. When all domains were discussed, the workshop moderators grouped the sticky notes, producing a heat map used to clarify priorities.
Citizens also filled a questionnaire about their experience of the panel.
Results
During the workshop, citizens listed 249 suggestions of health system performance measures: 153 marked as extremely important, 79 as important, and 17 as somewhat important. The domains with most measures proposed were related to people-centeredness, coordination of care, coverage, health and well-being, and equity.
Data analysis and consolidation enabled the researchers to compare the citizens’ ranking of themes with that of the other panels. These panels followed a similar structure to that of the citizen panel.
While the ranking of topics was similar, the study revealed a great variation in the priority assigned to each domain by the citizen panel compared to the other panels. Distinctively, the top-ranking domains for the citizens (people-centeredness, coordination of care, and coverage) were less prioritized by the internal and external stakeholder panels. Conversely, themes that were less prioritized by the citizen panel, like accessibility, responsiveness, efficiency, and effectiveness, were prioritized more in the other panels.
Conclusions
The key strengths of the study are the participatory and inclusive characteristics of the process of co-producing the HSPA framework by including three stakeholder panels. However, using only one citizen panel does not permit to achieve representativeness of Ireland’s population.
In this study, citizen panellists shared a common understanding of what a citizen panel involves. Most came well prepared, were fully engaged in the discussions and considered their participation as part of their civil duty, in terms of representing other citizens.
The citizen panel prioritized more people-centeredness aspects of performance domains compared to the internal and external stakeholder panels. This prioritization by the citizen panel is aligned with the premise of the need the health system to achieve greater value, performance, and person-centred quality of care and the need to include citizens’ views to steer policies to better address the populations needs, expectations, and preferences.
The authors claim that the use of a citizen panel in the scope of the co-production of an HSPA framework shaped the processes that followed, with the restating of priorities by the citizen panel informing decision-making throughout. This approach, based on principles of inclusiveness in governance and in creating value in health care, contributed to fostering the legitimacy, responsiveness, and accountability of the framework in safeguarding the interests of a broader range of stakeholders.
The opinion of the SPX Virtual Contents Editorial Board:
« Beyond the participatory approach adopted in a government health policy program, by involving citizens in the prioritization of key areas, what seems particularly interesting to me in this article is the difference in perception between citizens and stakeholders (internal / external) involved in the health system. Not only does this once again underline the need to involve citizens / patients in discussions around the health system, to ensure that developments actually meet their expectations. But above all, we see that where the stakeholders willingly focus on structural aspects of the system, citizens are there to remind people that their priority is to have care thought out, planned and carried out around the patient as an individual. »
« This article analyzes the Irish initiative to involve patients in the configuration of a framework for improving the structure in the context of health system reform, through the Health System Performance Assessment (HSPA), which is a participatory process specific to the country, well established and widely accepted for evaluating the entire healthcare system. In 2019, researchers were commissioned to develop a results-oriented framework for the HSPA with multiple stakeholders including citizens. Including citizens makes the discussion focus on experiences that have value, it also makes them see that they belong to the system, and contributes to making care more patient-centered and prioritizing the care and results that matter to the patient.
This framework was built with the inputs of three panels, with professionals, professional associations, academies and patient organizations, and the third with patients whose representation was 15 people.
During the citizens’ workshop, preferences were collected on how the information should be measured and reported in a prioritization exercise. They were tasked with identifying the channels from which information should be reported to the public specifically on health and well-being, equity, accessibility, quality of care, social and financial risk protection, coverage, safety, responsiveness, efficiency, effectiveness, people -centered, continuity of care, and coordination of care.
Subsequently, the importance was assessed and later they were grouped to produce a priority map. The most prominent domains were those that referred to people-centeredness, coordination of care, coverage, health and well-being, and equity. The results were then compared with those of the other panels.
The interesting thing about the study is that although the ranking of topics was similar, the study revealed great variation in the priority assigned to each domain by the citizen panel compared to the other panel.
The top-ranking domains for the citizens (people-centeredness, coordination of care, and coverage) were less prioritized by the internal and external stakeholder panels. Conversely, themes that were less prioritized by the citizen panel, like accessibility, responsiveness, efficiency, and effectiveness, were prioritized more in the other panels. Thus, the conclusions are that since the citizen panelists prioritized the need for the health system to have more value, which is defined by who receives the benefit, not who gives it, and more quality of person-centered care. The fact of including citizens in the panel to produce the HSPA framework, focuses the priorities on the needs of the patients, and fosters the responsiveness of the framework, respecting all stakeholders. »
