Five-Star Hospital Compare: a one-star endeavor

Dec. 19, 2016 / By Richard Fuller, MS, Norbert Goldfield, MD

In this month’s blog we return to a recurring theme—the issues related to risk adjustment and performance measurement. More specifically, we think it is important to take a closer look at the Hospital Compare overall rating also known as the hospital “Star Ratings.” Concern with the Compare ratings have permeated the hospital industry from hospitals to MedPAC¹. The bulk of these concerns fall within three categories. The star ratings are:

• Biased against hospitals with a large percentage of low-income patients
• Do not represent hospital quality
• Confuse patients

Before addressing these concerns, it is worth summarizing exactly what goes into the Five-Star ratings.

As shown below, stars are awarded by assessing hospital performance across 64 measures subdivided within seven weighted categories.

Source: How the hospital overall rating is calculated; Medicare.gov

First, there is plenty of reason to expect bias against hospitals with greater shares of lower income patients. This simply follows the same scenario that is seen in CMS readmissions rankings—a measurement over which we have regularly highlighted concerns. For the star ratings to deliver something other than a repetition of these results would be surprising—and they do not surprise. According to a study released at the end of November² (and the impetus for this blog), 20 percent of variation in star ratings across cities is correlated with unemployment and poverty rate—plenty of confirmatory support for the suspected bias against hospitals serving low-income populations.

The concern that these summary rankings do not represent meaningful differences in quality goes further. We have previously outlined concerns over the CMS complication measures, particularly  inadequate/absent risk adjustment coupled with miniscule variation with undue weight on final rankings.  The CMS approach results in bias against hospitals taking on more complex cases. To this we can add concerns raised by others over CMS mortality measures³: that low-volume hospitals are inappropriately ranked. If you are following along, the CMS approach to performance ranking leads to bias against larger hospitals with more complex cases taking on larger shares of poorer people.

The measures themselves have raised MedPAC concern that quality differences between star ratings are not “apples to apples.” For example:

• hospitals with five stars are ranked far less frequently on the four outcome domains (57 percent) than one star hospitals (98 percent); and
• five-star hospitals (on average) have 36 percent of admissions through the ED while one-star hospitals receive 78 percent through the ED.

Lastly, there appears to be little rhyme or reason for the weight assignment to the different measures. Why 22 percent for readmissions? 22 percent for mortality? And, then all of a sudden, efficient use of medical imaging comes out of nowhere with 4 percent? There is also concern that the mathematical construct of the measures (the way in which measures have been selected and summed to provide a single ranking) is flawed or, at best, arbitrary⁴. Taking all these points together we get a sense of the magnitude of the concerns.

Which, of course, leads to the next challenge: the star ratings are confusing for patients. CMS provides guidelines for patients with the advice that, “Hospitals provide quality care when they give their patients the care and treatments known to get the best results for their condition.” This is good advice that seems hard to argue with. Within the same guidelines, patients with an emergency are directed to go to their nearest hospital.  This means that to the extent that star ratings are intended to steer patients, they can only do so for elective conditions for which treatment outcomes and hospital performance can be researched in advance. Thus, Hospital Compare is intended for that purpose; however, it greets users as they land with the, “overall star rating, summarizing up to 64 measures of quality shown on Hospital Compare. But this is clearly not a summary of how patients might expect the hospital to perform for their condition. As detailed in the summary table, the 64 measures represent metrics that CMS has recycled from other programs and reporting efforts, weighted within and across domains to generate a ranking. Outpatient imaging, ED wait times and the propensity for a hospital to perform C-sections is unlikely to influence a Medicare enrollee’s selection of “getting the best results for their condition.” Stars are, however, easy to interpret: more are better. The purpose of guiding patients to get the best results for their condition and the ratings are incompatible.

We would suggest that a better approach for a star rating system for hospitals is contained in the bipartisan Hospital Outcomes Act of 2016 (H.R. 6274). This bill requires that the assessment of hospital performance on readmission and complication rates be based on a comparison across a comprehensive list of potentially preventable readmissions and complications that are risk-adjusted using a categorical clinical model. While the bill is limited to complications and readmissions, it could easily be expanded to include other outcomes measures such as the rate of return ED visits following a hospitalization and post- acute care expenditures. A focus on the outcomes achieved provides a more meaningful measure of hospital performance than the current underpinnings to the Five-Star rating system –and this approach can actually help patients determine where to get the best results for their conditions.

Richard Fuller, MS, is an economist with 3M Clinical and Economic Research.

Norbert Goldfield, MD, is medical director for 3M Clinical and Economic Research.

References

1. Dickson V. MedPAC questions validity of CMS’ hospital star ratings. Modern Healthcare. 2016:http://www.modernhealthcare.com/article/20160927/N.
2. Hu J, Nerenz D. Relationship Between Stress Rankings and the Overall Hospital Star Ratings: An Analysis of 150 Cities in the United States. JAMA Intern Med. November 2016. doi:10.1001/jamainternmed.2016.7068.
3. Silber JH, Satopää VA, Mukherjee N, et al. Improving Medicare’s Hospital Compare Mortality Model. Health Serv Res. 2016;51 Suppl 2:1229-1247. doi:10.1111/1475-6773.12478.
4. Atkinson JG. An Analysis of the Medicare Hospital 5 Star Rating and a Comparison with Quality Penalties.; 2016. https://www.academia.edu/30384615/An_Analysis_of_the_Medicare_Hospital_5-Star_Rating_and_a_comparison_with_Quality_Penalties.