Liberty Place, Suite 700
325 Seventh Street, NW
Washington, DC 20004-2802
(202) 638-1100 Phone
Monday, December 2nd 2002
Kenneth W. Kizer, MD, MPH
President and Chief Executive Officer
National Quality Forum
601 13th Street, NW
Suite 500 North
Washington, DC 20005
Dear Dr. Kizer:
The American Hospital Association (AHA), on behalf of its nearly 5,000 member hospitals, health care systems, networks, and other providers of care, appreciates the opportunity to comment on the National Quality Forum’s (NQF) proposed second set of Hospital Core Performance Measures. This group of 17 proposed hospital core measures was identified through the workshop that was conducted in mid-October, shortly after the adoption of the first set of 31 measures.
We are committed to making performance data and other information about hospital quality available to the public, and we were pleased to be able to support nearly all of the measures in the original set of 31. As organizations that are committed to quality, America’s hospitals use many of the proposed measures in both the original set and this draft set to focus and track their own quality improvement efforts. We believe that most of the proposed measures are useful for such quality improvement. We are, however, concerned that some of the 17 measures being recommended for use in public reporting are not appropriate for that purpose. Some of the suggested measures need a great deal more testing and work before they would be appropriate for public reporting. We also have concerns about the process by which these measures have been brought forward.
Comments On The Proposed Measures
We are concerned that the document sent for member review contains no information whatsoever on the evidentiary base for the proposed measures or the experience organizations have in using the measures. It also does not include a description of the rationale for selecting the proposed measures. There are general descriptions of the gaps in the initial measure set that were noted by a large number of members of the NQF in commenting on the initial set of 31, but there is no indication of the rationale for why, specifically, abdominal aortic aneurysm was added as a priority area, or why any of these specific recommended measures should be added. In the NQF’s Consensus Development
Process document 1.5, which was approved by the Board in January of this year, the section on product development states that the NQF products will be “explicit about the scientific evidence and experience underlying the recommended indicators, the criteria for selecting them, and the rationale for recommending the particular measures or approach.” (Section III.E.) This information is vital for members and others to appropriately consider the proposed measures. We urge the NQF staff to redraft this document to include the specific evidence and rationale for each measure, and to then resend the document to members for review.
Five of the proposed measures derived from the Agency for Healthcare Research and Quality’s (AHRQ) Hospital Cost and Utilization Project (HCUP) project are not yet appropriate for adoption. They are: abdominal aortic aneurysm (AAA) repair volume; AAA repair mortality; carotid endarterectomy volume; pediatric heart surgery volume; and pediatric heart surgery mortality. The HCUP indicators were developed to help target quality improvement activities, not to be used for public reporting. AHRQ has contracted with researchers at Stanford to examine the usefulness and appropriateness of the measures for use by the public, and that report is expected very soon. We strongly suggest that NQF obtain the report from AHRQ as soon as it can be made available, share it with Steering Committee members, and determine whether any of these suggested measures should be withdrawn. For those that remain, information from the AHRQ report should be summarized and included in the description of the evidence for these measures.
Several of the proposed measures are volume measures, including the three HCUP volume measures, and coronary artery bypass graft volume, coronary artery bypass using internal mammary artery, and percutaneous coronary intervention volume. Volume measures are suggested here as a proxy for quality, but we have no evidence that they are an appropriate proxy. Since there are real outcome measures among the set adopted by the NQF, there is little reason to have proxy measures of quality. The relationship between volume and high quality outcome has long been asserted, but studies have shown mixed results. For some procedures, greater volume seems to be associated with better outcomes, but for others, no relationship has been found. For example, a 1996 paper by Rutledge, et al showed no relationship between volume and mortality for ruptured AAAs. Other studies have confirmed this lack of a relationship between volume and outcome for ruptured AAAs. The recent study by Birkmeyer and others in the New England Journal of Medicine showed an insignificant (0.2%) difference in mortality for carotid endarterectomy between high volume and low volume groups of hospitals. Even where there is a significant difference between the outcomes for the high volume group of hospitals and the low volume group of hospitals, there is no evidence that outcomes for a particular hospital can be predicted based on volume. We urge NQF to drop the volume measures from this list.
Among the patient safety measures, four are relevant for infection control: catheter associated urinary tract infections rates; urinary catheter use rates; central venous line associated infection rates; and central venous line use rates. These come from the national nosocomial infection surveillance system, and have been used to promote quality improvement in the participating hospitals. For these types of internal quality improvement practices, it is useful and appropriate to include the catheter and central line use rates, but for public reporting, they are likely to be difficult to understand and misleading. Hospitals use the information on catheter and central line use rates for quality improvement purposes because it has been shown that one of the most effective methods of preventing an associated infection is to minimize the number of days the catheter or central line is in place. However, without an appropriate risk adjustment method to account for the differences in patients and their need for catheters or central lines, this information would be misleading. We urge the NQF to retain the catheter associated urinary tract infections rates and central venous line associated infection rates as measures appropriate for public accountability, and to drop the catheter and central line use measures from this list.
Among the patient safety measures are two that are specified and measured strangely, and as a result might be misleading: restraint prevalence and falls. Restraint prevalence is specified as physical restraints only, counted on a single day during the year. It is not clear that a single day count will provide an appropriate representation of the general prevalence of restraints in a facility, nor does there appear to be any type of adjustment to account for differences in the patients being cared for that might appropriately lead to differences in the use of restraints. For example, hospitals that have a large psychiatric patient population may make greater use of restraints to prevent people during an acute episode from harming themselves or others. This is appropriate and understandable, but would not be recognized in the proposed measure. Additionally, by counting only physical restraints, this measure may have the unintended and undesirable effect of encouraging greater use of chemical restraints.
The safety measure for falls per 1,000 patient days is also problematic because of its definition. It appears to include patients who lose their balance in the presence of a nurse or other staff and are assisted to a safe landing by that staff member. Getting patients up and moving post surgery is important and productive for their recovery. Balancing the benefits of getting patients moving with the risk that they will be unsteady may result in some unintended descents to the floor, but as long as staff are taking appropriate cautions to prevent injury this should not be regarded in the same light as a patient falling unattended, when the likelihood of injury is far greater. Because of the problems with each of these measures, we urge that the restraint and fall measures be dropped and that the developers be encouraged to consider modifications to the definitions that would enable these measures to be used in the near future.
Comments on the Process
We have two process concerns about the handling of this second proposed set of hospital core measures. First, we are concerned by the use of a workgroup to review and recommend a set of measures to NQF members. According to the NQF Consensus Process, proposed standard measures are to be developed through a steering committee or review committee. We see no prescribed role for a workshop panel or group. Since the NQF had a steering committee to review and recommend measures for the hospital core set, it would have seemed appropriate and in keeping with the approved Consensus Process to have that steering committee review and recommend additional measures to the members rather than establishing a separate two-day workshop with other individuals serving as members of the panel. We hope that NQF will follow its own stated process in the future.
Second, the NQF has not yet adopted a framework for hospital quality measurement. A framework document was sent to members for review, and we understand that many comments suggesting numerous revisions were received. NQF staff is now reconstructing the draft framework based on those suggestions. It is counter-productive and confusing to try to determine specific measures before we have actually agreed on what the framework for those measures will be. We cannot know how these measures would help create a hospital measure set that is balanced, parsimonious, credible, useful, or in keeping with any of the other desirable characteristics of such a measure set. By selecting measures without the framework, we may be approving measures that add to the already substantial measurement burden hospitals experience, without giving potential users the information they really need. We urge the NQF staff not to send this second round of measures out for a vote until the Hospital Measurement Framework document has been revised, reviewed and voted on by the member of the NQF. Then, the second set of measures should be sent out with a table showing how each of the measures, along with the initial set, would fit within the framework. Since work on the framework document is well under way, we do not believe that waiting for its review would result in more than a 60-day delay in the process, a wait very much worth the substantial improvement that would result.
If you have any questions or concerns about our comments, please contact Nancy Foster, Senior Associate Director for Policy, who can be reached at 202-626-2337.
Executive Vice President