Researchers in the Division of Postgraduate Medical Education at Massachusetts General Hospital published a study, “Attendees’ perceptions of commercial influence in noncommercially funded CME programs“, in the Journal of Continuing Education in the Health Professions‘ 2012 Summer edition. The research was funded by academic institutional support from the Massachusetts General Hospital Psychiatry Academy.
The study had similar findings to similar studies that showed little to no commercial influence at similar levels to supported programs.
Introduction
The researchers outline the general arguments for and against industry sponsored CME programs. They note: “For those advocating for reduced involvement from proprietary entities in CME, the primary concern has been bias due to commercial influence … These assertions are based on the premise that proprietary entities would be unlikely to invest in CME programs unless there was a positive impact on sales.”
However, “there are concerns that terminating commercial support will deprive providers of critical information about industry products and science conducted in the private sector, and create barriers to participation if providers are forced to pay steep tuition. Proponents of commercial support highlight safeguards in Accreditation Council of Continuing Medical Education (ACCME) guidelines.”
The key question: “In this study, we sought to clarify what sources contribute to attendees’ perceptions of commercial influence in non-industry-supported CME programs, and how attendees perceive that this influence is manifested at both the speaker and program levels.”
Method
The study was separated into two phases. In the first phase, the researchers aimed to optimize the wording of the bias question to avoid ceiling effects and understand the perspectives of a larger sample of bias-perceiving attendees. They qualitatively derived specific sources and manifestations of attendees’ perceptions of bias from their free-text responses. In the second phase, they asked attendees the optimized question for each speaker as well as for the entire program. Sources and manifestations identified in Phase 1 were included in the second phase as “Check all that apply” options to prompt attendees to be specific about their perceptions of bias.
Both phases included only tuition-funded CME programs that received no commercial support. To encourage attendees to freely disclose their perceptions of speakers and CME events, all evaluation forms were anonymous and no demographic data were gathered. Both phases of the study were deemed minimal risk by the Partners Institutional Review Board and exempt from written informed consent requirements.
RESULTS
A total of 677 attendees responded in the Phase II evaluation (96.6% response rate). The majority (532; 78.6%) answered the question addressing commercial influence for the entire program. Of these, 1.7% responded “Yes,” 9.4% responded “Somewhat,” and 88.9% responded “No.” Across programs, a range of 6.0% to 9.3% of attendees rated the entire event as “Yes” or “Somewhat.” On average, attendees rated 57.1% of the lectures.
Asking about bias per individual lecture gathered indications of bias from more attendees than did asking about bias per entire program. The majority (74.6%) of attendees who answered the bias question for the entire program “Somewhat” or “Yes” also indicated that 1 or more individual lectures were biased. At two programs, 100% of attendees rating the entire program biased further identified specific lectures.
When attendees were stratified based on how they rated the entire program, there were significant differences in the percent of individual lectures that they rated biased. For example, on average, participants rating the entire program “Yes” rated a significantly higher percent of lectures “Yes” than did participants who rated the entire program “Somewhat”.
Seventy-seven (11.4%) attendees checked at least 1 source of bias, and 67 (9.9%) checked at least 1 manifestation of bias. Of the 77, 18 (23.4%) rated the program free of commercial influence. Significantly more attendees identified the funding of individual speakers as a source of bias than funding of the hosting institution. The rating of the entire program as “Yes” or “Somewhat” correlated with whether attendees considered funding of the research referenced to be a source of bias. In the researcher’s validity check, despite the fact that all were noncommercially supported programs, 19.8% of attendees identified the funding of the overall event as a source of commercial influence. The top 2 manifestations of bias—speakers’ mentions of particular pharmaceuticals or products and speakers’ expression of personal opinions about particular pharmaceuticals or products—were cited significantly more than any other manifestations.
DISCUSSION
The bulk of the following text comes from the researcher’s analysis of the study:
This study provides new insight into attendees’ perceptions of commercial influence in CME programs that are not commercially supported. Methodologically, it was necessary to ask attendees about bias per speaker as well as per program to fully capture their perceptions. Clarifying what they thought constituted bias, attendees most frequently reported perceiving speaker-level sources (eg, individual speakers’ funding and funding of the research referenced in their slides) and manifestations (eg, number of mentions and personal opinions about pharmaceuticals) of commercial influence.
There were several intriguing results, including the range of sources and manifestations of commercial influence perceived at noncommercially supported programs. Although policy changes have focused on CME program funding (and commercial support has decreased across CME organizations), 20 attendees perceived many additional sources and manifestations of commercial influence. It is possible that modifying evaluation forms to include the comprehensive questions described here would assist with detecting differences in perceived bias based on commercial funding, thus allowing for more rigorous assessment of commercial influence in CME programs.
Asking attendees about commercial influence in individual lectures and the overall program reflects best practices and provides a template for future CME evaluation forms. Sixty percent of bias ratings came from assessments of individual speakers. It is unlikely that these data would have been captured using previously published methods of asking only about the CME program as a whole.
Additionally, the 3-point answer format and precise wording of the question (asking whether the content was commercially influenced) differed from previous techniques. The importance of this 3-point answer format was emphasized, as the percent of lectures rated biased differed significantly based on attendees’ rating of the entire course using this 3-point scale. For example, attendees responding “Yes” to the question about the entire pro- gram rated significantly more lectures “Yes” than attendees who rated the entire program “Somewhat” or “No”.
With regard to the question wording, it is possible that the neutral tone of the question or asking about influence on course content, led to more indications of bias from attendees than did the general and negatively balanced question about inappropriate commercial pressure. While the higher rate of indications may be indicative of false positives, this wording could be useful in future studies comparing commercial and noncommercial CME, which have been hampered by a ceiling effect of attendees perceiving almost no bias regardless of commercial funding. Even with our more inclusive wording, few participants rated the entire program biased. Moreover, the response rate indicates that attendees were willing to respond to these questions.
The identification of specific sources and manifestations of commercial influence highlight the complexity of designing evidence-based CME programs. The importance of individual speakers’ funding for attendees’ perceptions of influence suggests that these ties to industry should be examined prospectively using risk estimate tools. While speakers’ references to research have been considered in the context of commercial influence, there are no specific standards in the ACCME Standards for Commercial Support that subjects them to scrutiny. Although this is an indirect path of commercial influence, if a speaker presents only studies that were funded by a particular institution or commercial supporter, the presentation is likely unbalanced. The researchers recommend that an external conflict-of-interest review by content experts consider the authors and funding of studies referenced in slides. Finally, while the number of mentions of pharmaceuticals/products is not currently regulated, this might also be considered in an external review, as it correlated with attendees’ ratings of entire CME programs.
While the data presented support the manifestations of commercial influence they also suggest notable exception. Previous research drew a distinction between a speaker’s bias and personal opinion, but attendees’ free-text responses suggest that both providing and withholding personal opinions can be indicators of bias. While noncommercial factors influence personal opinions, the demarcation between unscientific bias and the bias of clinical expertise is still undefined.
Limitations of the study include a small number of CME events, and that all activities were produced and hosted by the same institution and were focused on psychiatry. However, this narrow range also created a more controlled environment. It is possible that the presence of “Check all that apply” questions created a cueing effect and led attendees to endorse sources and manifestations of bias that they did not perceive. The researchers attempted to control for this by designing the options based on previous attendees’ free-text responses. Also, when ordered by frequency, Phase II sources and manifestations were very similar to those endorsed in Phase I, and the efficient “Check all that apply” format may have led to our higher response rate in Phase II.
In addition, more than 20% of attendees who identified sources and manifestations of bias did not rate the entire program as biased. They may still have perceived these sources and manifestations without finding the entire event influenced, or were answering the questions with regard to CME activities in general, as has been studied previously. Finally, self-report from attendees not specifically trained in identifying commercial influence limits the reliability and validity of their ratings, and such perceptions of bias may not indicate actual bias. As the CME field is concerned about the appearance of bias as well as actual commercial bias, these perceptions are worth understanding. The data suggests that if CME providers ask their attendees to specify individual lectures and particular sources and manifestations of bias, attendee ratings could provide a more valuable assessment.
The data presented suggest directions for future research into the question of attendees’ perceptions of commercial bias. The relationship between rating individual speakers biased and rating the entire event biased prompts further inquiry. Does a certain percentage of speakers have to appear biased before the entire event is characterized as such? An important next step will be to investigate whether specific sources or manifestations of commercial influence have an impact on patient care, as this would enable targeted remediation of CME planning. Additionally, this methodology could be used to explore noncommercial sources and manifestations of bias (eg, did the speaker reference systematic reviews or promote referrals to his/her practice?), providing a necessary context for commercial bias within CME.
With regard to the validity check, almost 20% of participants who identified any source of commercial influence selected “Overall event funding.” This could not be a source of commercial influence, as the only funding for these events came from attendee tuition. As commercial funding for CME events has been controversial as a potentially strong source of bias, it is noteworthy that attendees identified it as a source of bias even without funding. While this could indicate a lack of validity in attendee perceptions (which has implications for studies comparing attendee perceptions of commercial vs. noncommercial CME), there are other potential explanations. The controversy surrounding this source may have increased attendees’ likelihood of selecting this option. Other explanations include misunderstanding the wording (eg, thinking “overall” meant any commercial funding associated with the event), or assuming that the event was commercially supported because the option was included. Randomly selecting attendees to debrief about how they interpreted the “Check all that apply” options would help clarify. Future studies should consider the influence of attendee belief of commercial support on their ratings of commercial influence.
CONCLUSION
In summary, attendees comprehend a great deal more in the term commercial influence than simply the impact of industry support on a CME program. Asking attendees to rate bias per lecture and specify sources and manifestations of commercial bias clarifies their perceptions of commercial influence in CME. This study presents a thorough and measurable outline of sources and manifestations of commercial bias, as well as practical recommendations for measuring attendees’ perceptions of commercial influence in future CME events. Caution should be taken with policy changes affecting CME funding as we continue to distinguish sources and manifestations of commercial influence in both commercial and noncommercially supported CME programs.
Overall the study reinforces that there is little need monitor non commercially supported programs for speaker commercial bias
POTENTIAL LESSONS FOR PRACTICE
The potential for these methods to bias learners to look for commercial influence is a factor to consider before taking any of these measures.
1. Evaluation forms area critical opportunity to learn about attendees’ perceptions of commercial influence. They should be asked about this per individual lecture as well as for the entire program.
2. For attendees who endorse commercial influence, ask them to identify its commercial source and how they perceived it to manifest in the program.
3. Before the program, an external review by content experts for risk of commercial influence should consider the research referenced in speakers’ slides in addition to speakers’ personal connections to industry.