ProPublica’s attempt to create a searchable database of physician quality for the public, the “Surgeon Scorecard,” has run into some credibility issues. The Surgeon Scorecard was launched in July 2015 and is a database that catalogues and provides the public with individual surgeon’s death and complication rates for eight different medical procedures. ProPublica created this database to help patients determine which surgeons, not necessarily which hospitals, are the best at each particular surgery. This foray into transparency in the medical field has had its fair share of issues and is not currently recommended to be a serious source of information for patients.
The most recent set of problems is brought to light in a nineteen-page Rand Corp. report that takes issue with the Scorecard’s methodology. Rand finds the issues with methodology to be so serious that patients should not view the Scorecard as “a valid or reliable predictor of the health outcomes any individual surgeon is likely to provide.”
One of the issues Rand takes with the Scorecard is that the Scorecard “masks hospital-to-hospital performance differences, thereby invalidating comparisons between surgeons in different hospitals.” This data omission results in a misleading of patients in a “systematic, albeit unintended, fashion.”
Another issue with the Scorecard is the questionable “accuracy of the assignment of performance data to the correct surgeon.” Rand states that claims data submitted to Medicare is what forms the basis of the Scorecard, and that because claims data is notorious for being inaccurate in individual provider assignments, the reliability of the entire Scorecard is negated.
These latest issues compound with previously mentioned concerns about the chilling effects such improperly culled information can have on patient care. In early August 2015, we reviewed a report issued by the American College of Cardiology. That report found that in Massachusetts and New York – two states with mandatory public reporting requirements – patients with myocardial infarction were much less likely to undergo a percutaneous coronary intervention than patients in states without public reporting mandates. As we have previously written, “public reporting and efforts at increasing the transparency of surgical procedures have a laudable goal in mind of guiding patients to the best care. While it is easier to point out the flaws in a reporting system than come up with a perfect methodology, the public reporting of PCI study shows that true patient harm can result if the reporting metrics continue unchanged.”
Rand Corp. reminds patients to ask all prospective surgeons about the risks of any procedures they may be contemplating, and for hospitals to monitor the quality of work their employees are performing, including surgeons.
Rand issued some recommendations for ProPublica to take under advisement that would improve the reliability of the Surgeon Scorecard. One of those recommendations was to conduct a validation study that compared claims-based surgeon assignments with those derived from medical records to help determine the frequency of misattributed events. Rand also recommended ProPublica establish minimum acceptable thresholds for measurement reliability, as that is considered best practices and this omission appears to impose a high risk of misclassifying surgeons and misdirecting patients and other report users.
Additionally, to their credit, ProPublica has made one clarification and one correction since the Surgeon Scorecard has been released, which shows that they have been willing to make necessary corrections to make the Surgeon Scorecard a more reputable source for patient information.
It seems that as time goes on, the Surgeon Scorecard may be a reliable indicator and a benefit to patients nationwide; however, right now, with the current issues, the Surgeon Scorecard is not recommended to be taken seriously by patients.