Clinical practice guidelines have significantly grown over the last few decades, and more than 2,500 now can be found in the archives of the Agency for Healthcare Research and Quality’s (AHRQ) National Guideline Clearinghouse. “However, concerns have been raised about the reliability of many of these guidelines and the processes involved in their development.
To address these concerns, the Institute of Medicine (IOM) issued a new set of standards for clinical practice guidelines in March 2011 intended to enhance the quality of guidelines being produced. In promulgating these standards, IOM focused on eight areas:
1. Establishing transparency
2. Management of conflict of interest (COI)
3. Guideline development group composition
4. Clinical practice guideline–systematic review intersection
5. Establishing evidence foundations for and rating strength of recommendations
6. Articulation of recommendations
7. External review; and
8. Updating
Despite these new standards, a recent study published in the Archives of Internal Medicine, noted that no systematic review of adherence to such standards had been undertaken since one published over a decade ago. As a result, two reviewers independently screened 130 guidelines selected at random from the National Guideline Clearinghouse (NGC) website for compliance with 18 of 25 IOM standards.
The study found that “Clinical practice guidelines are falling far short of their mandate from the Institute of Medicine (IOM) that they be transparent and objective, as well as untainted by conflicts of interest, researchers reported,” reported MedPage Today.
Specifically, they identified inadequate information on conflicts of interest, which was provided for committee members in less than half of guidelines examined. Moreover, of those guidelines including such information, COIs were present in over two-thirds of committee chairpersons (71.4%) and 90.5% of co-chairpersons. Guidelines from non-U.S. groups and medical specialty societies were the most unlikely to include this information.
The median number of IOM standards met by a random sample of 114 sets of guidelines was eight out of 18 (44.4%), and fewer than half of the guidelines adhered to at least 50% of the standards, according to Philip A. Mackowiak, MD, of the University of Maryland in Baltimore, and colleagues.
“Subspecialty societies were the worst in this regard, with barely a third of their guidelines satisfying more than 50% of the IOM standards surveyed.” The noted the failure of guidelines from specialty societies to provide conflict-of-interest information, the investigators noted.
“Less than a third of guidelines prepared under the aegis of subspecialty organizations — whose recommendations carry added weight because of their special expertise and whose members stand to profit directly from such recommendations — included information on [conflicts of interest],” the researchers argued.
“Everybody everywhere is developing guidelines and there is no real quality control,” Dr. Mackowiak, told Reuters. “There is no good oversight of who actually develops the guidelines or what criteria need to be met in order for them to be published.”
Other important shortcomings were a lack of information on the process by which committee members are chosen, and non-inclusion of patient representatives or information scientists. “Of the specific areas in which clinical practice guidelines need to be improved, none is more pressing than that having to do with the composition of committees developing the guidelines,” the researchers stated.
In addition, benefits of treatment were more often emphasized than potential harms, and were only presented as generalizations. A further shortcoming was that differences of opinion on the published evidence were seldom presented. "Guidelines were nearly always written in such a way as to suggest that recommendations were unanimously supported by committee members," the researchers noted.
A final concern was the age of many guidelines. Some authors have suggested that guidelines should be updated every 5 years, yet fewer than half of those included in this review had been updated.
One area in which the guidelines appeared more satisfactory was in gathering and organizing evidence, although abstracts and publications in languages other than English were rarely included.
The researchers compared their findings with a similar report published in 1999, and concluded that “little, if any, progress has been made over the past quarter century in improving the quality of clinical practice guidelines.” An even harsher assessment was made in an invited commentary by the author of the 1999 study, Terrence Shaneyfelt, MD, of the University of Alabama at Birmingham.
“The same problems that have plagued guideline development continue to plague guideline development; namely, their variable and opaque development methods, their often conflicted and limited panel composition, and their lack of significant external review by stakeholders throughout the development process," Shaneyfelt stated.
In his commentary, Shaneyfelt argued in favor of the centralization of guideline development, to avoid contradictory recommendations that can only make clinical practice more difficult. He further stated that AHRQ should only publish guidelines that adhere to the IOM standards, but is “not optimistic.” “I hope that efforts by the Guidelines International Network are successful, but until then, in guidelines we cannot trust,” Shaneyfelt concluded.
The commentary and article fail to point out the value of guidelines and how their use has improved practice. Often one gets caught in the weeds of an argument and misses the forest of good that guidelines have created. No process will be perfect, the IOM has some very good ideas in their recommendations but finding experts who have not worked with the tools they are developing is not one of them.