Health system performance comparison: an agenda for policy, information and research (2012)

Key messages

Policy issue

• International health system performance comparisons have the potential to provide a rich source of evidence as well as policy influence.
• Country comparisons that are not conducted with properly validated measures and unbiased policy interpretations may prompt adverse policy impacts and so caution is required in the selection of indicators, the methodologies used, and the interpretations made.

Lessons from international comparisons

• International health system comparisons provided by multilateral institutions such as the WHO or the OECD have generated much interest since the publication of the 2000 World Health Report, which highlighted the potential for cross-country learning from the scrutiny of comparable data, and also the challenges that must be overcome:
• • Definitions of performance indicators should be clear and consistent, and fit into a clear conceptual framework.
  • The metrics used in international comparison should enjoy widespread acceptance, and are defined in unambiguous terms that are consistent with most countries’ data collection systems.
  • In order to draw meaningful comparisons and understand the drivers of differences in measures between systems it is usually necessary to adjust for variations in the demographic, social, cultural and economic circumstances of nations.
  • “Single number” measures of whole health system performance, while offering a more rounded view of performance, have limited scope for policy action, and may distract policy-makers from seeking out and remedying the parts of their system that require attention.
• Lessons from benchmarking activities in other sectors suggest that – when applied to health systems – benchmarking will be most effective if it focuses on practice as well as performance; is grounded in the broader change process; is well structured and planned in order to engage stakeholders; and carefully considers how performance is linked to resource allocation.

Lessons from recent experience in health system comparison

• In defining the boundaries of the health system it is important to be aware of the benefits of choosing both narrow and wide boundaries. Narrow boundaries are better suited to holding stakeholders accountable, while broader boundaries are better for a more holistic understanding of the determinants of valued outcomes.
• Performance measurement evaluates the extent to which a health system meets its key objectives. However progress in the development of data collection techniques in the different dimensions of health performance is variable.
• While efforts should be undertaken to improve data collection efforts, policy-makers should also make themselves familiar with limitations in existing indicators in order to be able to interpret them appropriately.

Comparing key domains of performance

• Population health measures often take a broad perspective, which captures the effect of many determinants of health beyond the delivery of health care. This broad perspective can be attractive from a political point of view because it draws attention to the importance of many sectors in determining health outcomes. However it also creates major methodological challenges in seeking to attribute changes in health to any particular actions. A narrower perspective, such as the concept of avoidable mortality, focuses on measures that can more easily be attributed to health care, and which are therefore more amenable to immediate health policy.
• All population health indicators suffer from a number of methodological problems, which need to be addressed in order to make international comparisons more meaningful. Some of the main issues involve availability and coding of data, particularly of data on cause of death where there are problems of comparability among countries and over time.
• Direct indicators of the contribution of health services to health status are available in the form of health service quality measures, such as standardized hospital mortality rates and numerous disease-specific health outcome measures, such as mortality rates, adverse events and complications. Far less prevalent are broader outcome measures in areas such as disabilities and discomfort.
• While existing health service measures offer some indicators of the performance of individual organizations, international comparison is complicated by different organizational settings and reporting conventions, even after suitable adjustment for case-mix and other contextual circumstances.
• Although comparative indicators on inequality of health and equity in access to health care are available at both European and non-European level, equity indicators derived from existing projects and datasets may be misleading for policy-makers due to limitations in the availability and comparability of data.
• The many aspects of financial protection have yielded some important indicators, such as the incidence of catastrophic expenditure. However it has proved difficult to develop a single indicator capturing the full extent to which people are financially protected from health shocks. Currently, measures of the incidence and magnitude of households’ direct payments for health care form the basis of metrics for financial protection assessment and system comparisons.
• There is still lack of clarity as to what dimensions should be included in the domain of responsiveness, which embraces concepts such as respect, confidentiality and prompt attention. This uncertainty leads to the measurement of different areas using different tools with different weights, domains and indicators that are difficult to summarize and compare.
• Efficiency indicators serve as a summary measure of the extent to which the inputs to the health system, in the form of expenditures and other resources, are used wisely to secure the goals of the health system. Almost all efficiency indicators are constructed as a ratio of inputs to outputs, offering an indication of the extent to which resources have been wasted along some or all of the production pathway.
• In measuring efficiency, a fundamental challenge is the assignment of inputs and associated costs to specific health system activities, often relying on arbitrary accounting rules or other questionable assignments. In principle, the inputs used should be directly and fully aligned with the output under scrutiny.

Future directions

• The key requirements for creating comparable indicators that address the needs of policy-makers are: appropriate methods of summarizing complex information; a narrative that picks out the key issues and uncertainties; a diagnosis of why the reported variations are arising; and an assessment of the implications for policy action.