Re: Nursing Home Performance Measures Project

Liberty Place, Suite 700
325 Seventh Street, NW
Washington, DC 20004-2802
(202) 638-1100 Phone

Thursday, August 8th 2002

Barbara Paul, MD
Quality Measurement and Health Assessment Group
Office of Clinical Standards and Quality
Centers for Medicare & Medicaid Services (CMS)
Sent via email to

Dear Dr. Paul:

On behalf of the American Hospital Association’s (AHA) 5,000 hospital, health system, network and other health care provider members, we thank you for expeditiously posting the report entitled, "Validation of Long-Term and Post-Acute Care Quality Indicators" per your previous commitment.  AHA has briefly reviewed the report and we believe that it contains some important conclusions.  We have three main concerns with respect to the public reporting of the post-acute care quality measures: bias in the measures resulting from the MDS data collection cycles, the importance of the Facility Adjustment Profile (FAP) in risk adjustment and the potential for unstable measurements resulting from the very low prevalence of the delirium measure. 

AHA’s membership includes about 2,000 hospital-based skilled nursing facilities (SNFs). We support the overarching goals of the nursing home performance measures project.  We believe that meaningful, accurate performance data can be used to improve the delivery of care and assist the public in making more informed decisions about their post-acute and long-term care choices.  Unfortunately, as we noted in our comments to the National Quality Forum (NQF), we believe that the post-acute care measures are “fatally flawed” and recommend their deletion from the pilot.  Nothing in the validation report changes our view.

One of the main concerns expressed in AHA’s comments to the NQF was about how bias could occur in the measures due to the way data is collected on the MDS for post-acute patients and the effects of systematic exclusion of successfully treated short-stay patients. Findings in the validation report bear out this concern.  For example, the post-acute measure “failure to improve respiratory problems” has a 92 percent prevalence rate.  This very high prevalence is due to the fact that the only post-acute patients being measured are those with long enough stays to have two assessments – these are, by definition, the less successful patients.

Based on our review of the validation report, AHA has additional concerns about the post-acute quality measures.  In particular, two of the three measures used in the six-state pilot (improvement in walking, pain management) were found valid only if risk adjusted by the FAP (see p. 51 of the validation report).  In general, this finding is not surprising as it makes sense that facility admission tendencies are likely to have much more of an effect on post-acute care patients than long-term care residents.  The AHA is concerned that CMS might broadly apply the report’s findings about the FAP, which will have the effect of invalidating these two post-acute measures.

The report’s findings regarding “failure to improve delirium” also raise a concern. It is described as a low prevalence indicator (3% prevalence).  The researchers raise questions (see p. 33 of the validation report) about the possible effects of very low or very high prevalence on the stability of the measures and their ability to discriminate among facilities.  Accordingly, this measure is a candidate for deletion.  

Since the pilot measures are the key measures being considered for the national implementation in October, the above concerns, at a minimum, need to be addressed prior to moving forward.  We continue to suggest, however, that CMS exclude the post-acute measures in the initial phase of the public reporting pilot due to the fundamental measurement problems cited above.

Finally, we question the report’s finding that the MDS data is sufficiently reliable for the purposes of public reporting.  We note that the inter-rater reliability findings in this validation report appear to be more optimistic than other independent assessments of MDS item reliability (see “Final Report on Assessment Instruments For PPS” by Joan Buchanan, PhD, February, 2001).  We urge CMS to continue to seek independent confirmation of the reliability of data derived from assessment instruments.

Thank you for consideration of these comments about this critically important project.  Please do not hesitate to contact me at (202) 626-2320 or Nancy Foster at (202) 626-2337 if you have any questions about these comments. We would like to speak with you at your earliest convenience to discuss this matter.


Brian Ellsworth
Senior Associate Director, Policy
American Hospital Association 

Cc:       Paul Elstein
            Yaell Harris


About AHA


Member Constituency Sections

Key Relationships

News Center

Performance Improvement

Advocacy Issues

Products & Services


Research & Trends


155 N. Wacker Dr.
Chicago, Illinois 60606

800 10th Street, N.W.
Two CityCenter, Suite 400
Washington, DC 20001-4956