Liberty Place, Suite 700
325 Seventh Street, NW
Washington, DC 20004-2802
(202) 638-1100 Phone
Wednesday, May 15th 2002
Kenneth Kizer, MD, MPH
President and CEO
National Quality Forum
Attn: Nursing Home Performance Measures Project
601 Thirteenth Street, NW
Suite 500 North
Washington, DC 20005
Dear Dr. Kizer:
On behalf of the American Hospital Association's (AHA) 5,000 hospital and health system and other provider members, we are pleased to submit these comments as part of the pre-voting review process for the Nursing Home Performance Measures. Our membership includes about 2,000 hospital-based skilled nursing facilities (SNFs). The AHA supports improving the delivery of care through efforts that are based on meaningful, accurate and measurable outcomes decided upon in a collaborative framework. We actively participated in the National Quality Forum's (NQF) Nursing Home Performance Measures' Steering Committee that was convened to develop a core set of performance measures for the purposes of public reporting and quality improvement.
Overall concern - inadequate time to develop and validate the quality measures
While America's hospitals support the overarching goals of the nursing home public reporting pilot project, we believe that current selected measures of quality are not yet adequately developed or ready for public view. Unfortunately, the Centers for Medicare & Medicaid Services (CMS) imposed an artificially short deadline of October 1, 2002 for nationwide implementation of the public reporting of nursing home measures. This created pressure on the NQF Nursing Home Performance Measures Steering Committee to rush its selection of quality "measures" for CMS, resulting in inadequate time to validate the measures themselves or the equally important process of adjusting them for differences in SNF caseloads.
The lack of further development and validation raises two risks. One, consumers may be confused or misled by the information. And two, some SNFs that admit sicker patients could actually be unfairly labeled as poor quality providers.
The artificially short timeframe imposed on NQF by CMS also led the Steering Committee to narrow its recommendations to readily available information. While the Steering Committee report discusses other dimensions of quality of nursing home care (e.g., patient satisfaction and quality of life), it offers no tangible recommendations of measures in those areas. This reinforces our basic concern that there needs to be further applied research, development and validation before we can assure the public that we are measuring what we purport to measure. Without these other important dimensions of quality, we face a fundamental threshold question: do the clinically-based measures that are being put forward adequately and accurately portray the quality level of a given nursing home (even if the technical issues raised herein are all adequately addressed)?
Several years ago, CMS commissioned a large-scale study to develop and validate the next generation of quality measures for long-term and post-acute care. The project, known as the "Mega QI (quality indicator) Project," will likely report important findings from a broader validation study of nursing home quality measures this summer. Unfortunately CMS, and the NQF by extension, did not wait for these important results before moving forward with national implementation.
Our overall concerns are shared by others, including independent experts that NQF relied upon as part of the process of reviewing the risk adjustment methods for this project. Attached is an excerpt from a report by Andrew Kramer, M.D., Division of Health Care Policy and Research, Professor of Medicine, University of Colorado Health Sciences Center, entitled "Report to the National Quality Forum Nursing Home Steering Committee on Quality Measures and Risk Adjustment Methods for Consumer Reporting of Nursing Home Quality," along with some additional comments by AHA. In the preamble to his report, Dr. Kramer states:
My recommendation to the NQF steering committee is that they take a very strong stance against public reporting of nursing home quality indicators until these issues are addressed because the indicators have a significant potential to mislead the public, whom we are trying to serve…
The issues Dr. Kramer referred to were the need to validate the quality indicators and associated risk adjustment approaches. Since the NQF relied upon Dr. Kramer's advice concerning risk adjustment, we ask that you also heed his advice concerning the larger issue of these measures' readiness.
Specific issues for hospital-based skilled nursing facilities
Of particular concern to hospital-based skilled nursing facilities are the four post-acute measures in the pilot - pain management, delirium, improvement in walking and re-hospitalizations. Many hospital-based SNFs provide care to medically complex patients for a short duration (often less than 14 days) after a hospitalization. Due to limitations in the data source - the Minimum Data Set (MDS) - information concerning these four post-acute measures for successful, short-term skilled nursing facility discharges are not recorded in the data.
This occurs because the MDS was not designed for outcome analysis in post-acute care. As a result, the assessment cycles for the MDS do not match the admission/discharge profile of many post-acute care cases. Patients who are successfully treated in a SNF and discharged home within 14 days of admission are not recorded in the post-acute quality indicators due to the structure of the MDS data collection cycles. This can create a potentially misleading view of a skilled nursing facility's actual quality of care since only the longer-staying post-acute care patients are recorded in the database. We believe this is a fatal flaw in the four proposed post-acute measures.
Finally, AHA is also specifically concerned about the re-hospitalization measure proposed for post-acute care. CMS was unable to move forward with this measure in the six-state pilot due to data integrity problems with the MDS. Because of these problems, CMS instead examined claims data to calculate the measure, which also raises concerns since such data has never before been used for this purpose. To our knowledge, these data quality issues have not been resolved. Moreover, the need to appropriately risk adjust this measure is absolutely essential, as it is likely that those facilities with high re-admission rates are caring for a medically complex caseload. We believe these issues must be resolved before moving forward with this measure.
The American Hospital Association looks forward to continuing to work with CMS, the Quality Improvement Organizations and the National Quality Forum to promote quality improvement in nursing homes and address our concerns. If you have any questions about these comments, please call Brian Ellsworth, Senior Associate Director, Policy, at (202) 626-2320.
Executive Vice President
Report to National Quality Forum Nursing Home Steering Committee on
Quality Measures and Risk Adjustment Methods for Consumer Reporting of Nursing Home Quality
Andrew Kramer, MD
Head, Division of Health Care Policy and Research
Professor of Medicine
University of Colorado Health Sciences Center
February 14, 2002
With Commentary By The American Hospital Association (AHA)
(AHA Comments Underlined)
Dr. Kramer: The pilot validation study on the Mega QIs [quality indicators] was insufficient to provide convincing evidence that these risk-adjusted quality indicators accurately identify facilities with quality of care problems.
The pilot validation study represents the only attempt to verify whether the risk-adjusted quality indicators proposed for national reporting identify facilities with quality of care problems. As the developers of the quality indicators have stated, a more stringent standard is required for public reporting in contrast to internal continuous quality improvement or even use in the survey process because the assumption will be made that these quality indicators are a true reflection of care problems in facilities. The pilot validation study is the only study that attempts to test whether the quality indicators identify poor performing facilities. Unfortunately, the researchers were faced with both time and sampling constraints. The report on this validity study is brief, making it difficult to assess all of the methods. Based on the information provided, however, I am not convinced we can draw inferences about the validity of the quality indicators from this study…
AHA comment: We strongly agree with Dr. Kramer and the CMS-funded researchers developing the "Mega QIs" that the standards for public release are higher than for other purposes, such as internal quality improvement. Currently, the proposed measures, especially those for post-acute care (see comments below), do not allow consumers to draw accurate inferences about the quality of care in SNFs.
Dr Kramer: The other type of evidence that was presented was not adequate to demonstrate that either of the risk adjustment methods are valid.
…In summary, without a comparison between the risk-adjusted quality indicators and quality of care in a sufficiently large sample of facilities, we cannot evaluate the appropriateness of either risk-adjustment strategy [stratification vs. facility-level adjustment]. These two risk-adjustment strategies, plus others, could be tested relative to this gold standard quality assessment with a particular focus on the facilities that change from unadjusted to adjusted and a thorough analysis of false negatives and false positives. While interesting, these other forms of examining distributions and changes as a result of risk adjustment do not substitute for a rigorous validation process.
AHA comment: Since the NQF Steering Committee on Nursing Home Measures' recommendations are so heavily tilted towards clinical measures, appropriate risk adjustment is crucial. Otherwise, facilities that admit medically complex patients will be unfairly portrayed to the public. We agree with Dr. Kramer that risk adjustment methods should be validated relative to a carefully constructed gold standard. This is a challenging task, but one which could be achieved with the proper amount of time and resources. Until such validation occurs, we can only guess that the risk adjustment methods employed are adequate.
Dr. Kramer: The post-acute care quality care indicators are influenced by selection bias due to attrition.
…I have serious concerns about using the proposed post-acute care quality indicators for public reporting. The problem is that only about 60% of SNF admissions have a 5-day and 14-day assessment available on which to base these quality indicators…the problem for the post-acute care QIs [quality indicators] is substantial.
The focus of the post-acute care QIs is the first 14 days, utilizing the 5-day and 14-day assessment pair. If 40% of the individuals admitted under SNF-PPS [prospective payment system] do not have these assessment pairs, then the quality indicator sample is susceptible to substantial bias…
…Even when these rates are risk adjusted using 5-day MDS [minimum data set] information on RUG [resource utilization group] category, diagnoses, functional status, cognition, etc. facilities vary substantially in community discharge rates. Thus, this quality indicator could yield better scores in facilities that are less likely to rehabilitate people to the point of community discharge early in their stay….
…A different type of problem can occur in relation to the PAC [post-acute care] QIs of failure to improve delirium symptoms and inadequate pain management/post-acute care. Facilities with a high rehospitalization rate because they have difficulty managing delirium, severe pain, or any other complex condition, would rehospitalize residents with the most severe problems that they were struggling to treat. A facility that manages even these more difficult cases with some success might keep these residents that are still somewhat symptomatic with respect to delirium or moderate pain, which would show up on their 14-day assessment. The dilemma is that in facilities where these problems deteriorated to the point where the staff could not manage them any further and residents were hospitalized within 14 days, a lower prevalence of symptoms in these QIs would be found on the 14-day assessment….
AHA comment: The MDS was not designed for outcome analysis in post-acute care. As a result, the assessment cycles for the MDS do not match the admission/discharge profile of many post-acute care cases. Patients who are successfully treated in a SNF and discharged home within 14 days are not recorded in the post-acute quality indicators due to the structure of the MDS data collection cycles. Dr. Kramer correctly points out that this may lead to bias in the reported findings. We believe this is a fatal flaw in the post-acute measures.