JRSM >
Volume 95, Number 10 >
Pp. 508-510
Centre for Health Economics, University of York, UK
1 Department of Management, University of St Andrews, Scotland, UK
2 Department of Health Sciences, University of York, UK
Correspondence to: Professor HTO Davies, Centre for Public Policy and Management, University of St Andrews, St Andrews KY16 9AL, Scotland, UK E-mail: hd{at}st.and.ac.uk
 | INTRODUCTION |
|---|
 TOP INTRODUCTION WHAT SHOULD BE MEASURED?ADVANTAGES OF OUTCOME MEASURESADVANTAGES OF PROCESS MEASURESHOW SHOULD WE MEASURE...REFERENCES |
|---|
| WHAT SHOULD BE MEASURED? |
|---|
|
TOPINTRODUCTIONWHAT SHOULD BE MEASURED?ADVANTAGES OF OUTCOME MEASURESADVANTAGES OF PROCESS MEASURESHOW SHOULD WE MEASURE...REFERENCES |
|---|
However, if the focus is on identifying and remedying apparent variations in performance, it is often preferable to measure not only outcomes but also the desirable processes of care. These can be viewed as professional actions recommended as good practice on the basis of expert opinion or evidence. From a performance management perspective, the key issue is that a desirable process should be unambiguously associated with improved patient health outcomes6. Monitoring the process can then be a substitute for measuring outcome. What are the advantages and disadvantages of process and outcome measures?
| ADVANTAGES OF OUTCOME MEASURES |
|---|
|
TOPINTRODUCTIONWHAT SHOULD BE MEASURED?ADVANTAGES OF OUTCOME MEASURESADVANTAGES OF PROCESS MEASURESHOW SHOULD WE MEASURE...REFERENCES |
|---|
Outcomes are often long-term in nature. Managerial attention to outcomes encourages providers to adopt technologies (such as health promotion) that recognize long-term benefits, rather than myopic ‘patch-up’ technologies that ultimately yield worse results.
Some measures of outcome are generic, and once implemented are likely to require only refinement rather than wholesale review. Almost all measures of process are specific to the technology used, and may become obsolete. Some aspects of outcome measures—most notably mortality—are relatively immune to manipulation by providers (although clinicians may be able to influence risk-adjusted outcome measures by exaggerating the risk characteristics of their patients)7. In contrast, many measures of process must rely on self-reported activity on the part of clinicians, and are therefore vulnerable to misrepresentation.
| ADVANTAGES OF PROCESS MEASURES |
|---|
|
TOPINTRODUCTIONWHAT SHOULD BE MEASURED?ADVANTAGES OF OUTCOME MEASURESADVANTAGES OF PROCESS MEASURESHOW SHOULD WE MEASURE...REFERENCES |
|---|
In some services there may be no consensus regarding what constitutes desired outcome (for example, in much of primary care). Even when there is general agreement regarding the appropriate concept of outcome (for example, mortality after surgery), agreeing a measure of the concept may be difficult. How long after surgery should outcome be measured? Should the quality of survival be measured? Even if these issues can be resolved, collecting the agreed outcome measure may be complex and costly.
Many outcomes of care become evident only after much time has elapsed. In long-term follow-up of patients the number of ‘missing’ observations may be very high, particularly amongst patients with adverse outcomes. Furthermore, reporting of adverse outcomes may be the responsibility of the very clinicians whose performance is being assessed, creating an incentive for under-reporting. Even if accurate measures of outcome can be assembled, their dissemination may be too late to influence clinical behaviour6.
Measures of process are usually readily attributable to the provider of care, and so are easily interpreted. By contrast, outcome measures are commonly open to challenge10: for example, they tend to be influenced by factors other than healthcare intervention (such as patient compliance and social circumstances), and so may require adjustment before any judgments can be made about professional performance. Outcome measures such as surgical mortality rates are often insensitive to the quality of healthcare received, or display a lot of random noise11. They are therefore often difficult to interpret with low volumes of patients (say at individual surgeon level or even unit level).
The philosophy underlying the production of clinical guidelines and pathways of care is that there exist readily monitored processes that lead to desired outcomes. Use of process performance measures is consistent with this philosophy. With appropriate use of information technology, many process measures such as patterns of drug prescribing can be made available rapidly, and unusual performance can be identified and acted on quickly if necessary. Poor performance on a process measure gives a clear indication of the remedial action required, whereas poor performance on an outcome measure gives no such direct guidance12. Often, therefore it is easier to devise incentive schemes associated with process.
In practice, both processes and outcomes matter to patients. Good performance management systems should therefore include measures of both. Moreover, the division between process and outcome is rarely as stark as suggested here. There is often a spectrum between immediate process and eventual clinical outcome, covering various measures of intermediate outcome. For example, the quality of clinical management of high blood pressure could in principle be measured by process (prescribing of appropriate drugs) or outcome (long-term quality-adjusted survival). However, a good intermediate measure of success would be maintenance of blood pressure within acceptable bounds.
| HOW SHOULD WE MEASURE PERFORMANCE? |
|---|
|
TOPINTRODUCTIONWHAT SHOULD BE MEASURED?ADVANTAGES OF OUTCOME MEASURESADVANTAGES OF PROCESS MEASURESHOW SHOULD WE MEASURE...REFERENCES |
|---|
Clinical teams
It is not always clear what lies within the control of the team. Suppose a
surgical team is grappling with a high rate of complications caused by
hospital acquired infections. If these complications are taken into account,
the team might seem to be doing well. But if the incidence of infection lies
within the team's control, performance should be assessed on the basis of
patient characteristics on admission.
Care is often delivered by more than one team. Even within one hospital, surgical outcomes might be the result of collaboration between surgical, medical and rehabilitation teams. Factors outside the institution, such as the referral practices of general practitioners, will also influence the outcome of care received in the hospital setting, making the performance of teams even harder to compare.
Patients
A fundamental requirement for comparison of performance is that differences
in the type of patients treated must be allowed for; but this rudimentary
insight has been widely ignored. An example is the use, as a clinical
indicator, of non-emergency deaths in hospital within 30 days of
surgery13. To
account for differences in diagnoses, severity and complications, risk
adjustment procedures have been developed in many specialties. In
intensive care the APACHE system is commonly
chosen14. Many risk
adjustment systems are at an early stage of development, and competing
mechanisms sometimes yield different
results15. The
challenge is to encourage those specialties currently with access only to
rudimentary risk adjustment mechanisms to develop more sensitive and
meaningful instruments.
The institution
The institution in which a team is operating can have an important
influence on outcome. This is particularly true in a hospital setting, where a
team may have to accept certain arrangements as immutable. One direct
influence is the level and nature of resources made available to the team.
Other factors include the extent and seniority of support staff, the
availability of theatre resources and the physical layout of facilities.
The external environment
Clinical outcomes are influenced by numerous local factors outside the
immediate institutional setting. These include geography and transport
infrastructure (affecting physical access to healthcare), arrangements for
social care, employment patterns and the social fabric. Most of these are
beyond the control of the clinical team, so, in principle, these should be
accommodated in any interpretation of measured performance.
Random fluctuation
In addition to the systematic influences on performance, there are
always random variations which persist even after adjustment for each
of the influences on outcome mentioned above. By definition, these random
fluctuations in measured performance are beyond the understanding or control
of healthcare professionals. They should nevertheless be properly taken into
account when differences in measured performance are being interpreted. Best
practice in the measurement of comparative performance will therefore always
entail reporting of confidence intervals in some form.
Numerous analytic techniques have been developed to address the ‘attribution’ problem. These come under the general rubric of ‘risk adjustment’, which embraces procedures as diverse as control charts, Bayesian techniques, multilevel modelling and countless multivariate statistical approaches16,17,18,19. As so often in healthcare, the appropriate approach is highly contingent on the purpose of the analysis, the nature of the data and the nature of the clinical specialty. There are furthermore important choices about how best to present analytic results to clinical professionals.
Good performance measurement is clearly an essential prerequisite of high-quality healthcare. In this paper we have argued that the choice of what to measure and how to measure have to be addressed carefully if the potential of performance measurement is to be fully realized. In part 2, we discuss ways to avoid the pitfalls of measurement.
| Acknowledgments |
|---|
| REFERENCES |
|---|
|
TOPINTRODUCTIONWHAT SHOULD BE MEASURED?ADVANTAGES OF OUTCOME MEASURESADVANTAGES OF PROCESS MEASURESHOW SHOULD WE MEASURE...REFERENCES |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  D. Nathwani From evidence-based guideline methodology to quality of care standards J. Antimicrob. Chemother., May 1, 2003; 51(5): 1103 - 1107. [Full Text] [PDF]
|
|
|
|
 |
|
 M. Goddard, H. T O Davies, D. Dawson, R. Mannion, and F. McInnes Clinical performance measurement: part 2--avoiding the pitfalls J R Soc Med, January 11, 2002; 95(11): 549 - 551. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
