The Good CME Practice Group (composed of medical education companies in Europe) published its core principles to drive standards for CME in Europe (see full description below); these relate closely to the new UEMS criteria:
- Appropriate education is based on clear learning objectives derived from identification of performance gaps and unmet needs.
- Balance needs to be evident in content, faculty and review, with content developed independently of sponsor.
- All relevant information should be disclosed to the learner, including how the content has been developed and the terms of financial support.
- Post-activity evaluation should measure satisfaction, knowledge uptake and intent to maintain or change behavior in line with learning objectives.
Pharmaceutical and device companies are now subject to more rigorous guidelines for support of CME events in Europe. These and other changes also affect managers of venues for these events, as was pointed out at the recent congress of the International Congress and Convention Association (ICCA), held in Puerto Rico. Coupled with the new criteria for CME accreditation, the coming year promises to be a learning experience for industry and congress directors. Why?
The European Medical Technology Industry Association (Eucomed) has just implemented a mandatory assessment of all European conferences in which a Eucomed member is participating, to determine if the event is compliant with Eucomed’s Code of Ethical Business Practices. John McLoughlin, chairman of the independent compliance panel, told attendees at the ICCA congress that a company can be sanctioned if the conference fails to meet assessment criteria including:
- The conference program must be focused on serious medical content targeted to attendees, have no gaps for recreation and be provided 90 days in advance for review.
- The location and venue must be accessible, conducive to a scientific session, not in a resort setting.
- Hospitality must be modest, not include entertainment, and include family or guests only at extra charge.
Guidelines for pharma are similar in some respects, much more detailed — but voluntary — said Martin Jensen of Lundbeck, vice-president of the International Pharmaceutical Congress Advisory Association (IPCAA) at the ICCA meeting. These went into effect early in 2012, and include:
- Promote scientific education and networking in a compliant environment
- Ensure than congress-related activities create a value for patients
- Societies must disclose budgets in advance and provide an audit after the event
- 5-star resorts are unacceptable, even if unrated
“The large European specialty congresses of today will shrink in size in the future,” Lew Miller of WMGS predicted. “Cost and advanced technology will move these events to be distributed in a number of smaller locations, linked by streaming video with audience interaction.” He also indicated that the trend toward performance evaluation of CME could change the nature of large didactic conferences.
CME Core Principles
CME in Europe is evolving rapidly with an increasing number of countries adopting voluntary or mandatory systems of CME participation for their physicians as evidence mounts that good CME programs improve clinical practice and outcomes. The requirements of educational programs are also changing rapidly, particularly with a need for clear demonstration that a CME-accredited activity has been developed independently by a faculty, meets educational needs, is free from bias and is of high quality. However, during this rapid evolution, it has become clear that there are misunderstandings and confusion amongst CME providers in relation to standard and regulations.
To address this challenging situation, the Good CME Practice Group undertook an initiative to establish a set of standard core principles with a view to adoption by European CME providers and other key organizations involved in provision of CME programs. The Good CME Practice Group consists of 12 European non-society education providers with shared objectives.
The Group developed four core principles relating to (a) appropriate education, (b) effective education, (c) fair balance and (d) transparency. In order to seek advice and input from peer groups and others involved in CME including accrediting bodies and medical societies, 93 representatives from these bodies were asked to complete a questionnaire and provide comments on the core principles. To ensure the final guidelines developed would be of highest relevance to the audience, thus promoting the highest chance of adoption, the following provisions were incorporated:
- To liaise with key stakeholder groups in European CME at each stage of development of the core principles to ensure alignment with their regulatory, legal and operational expectations. These stakeholders comprise members of the CME accreditation bodies, medical societies and pharmaceutical company financial supporters.
- To validate the guidelines through a formal consultation process with the ultimate target audience (through the European and national medical societies) and the CME accreditation bodies themselves.
- Ensure that all recommendations are clear and practical in today’s CME environment
Consultation phase
Drafts of the core principles were tested during a consultation phase with key stakeholders in European CME. The consultation involved participants from a range of organizations including national and European accreditation boards and medical societies across Europe. Participants in the consultation process were asked to comment on each individual principle in terms of whether it would set an appropriate standard, whether implementation of the principle is feasible and whether application of the principle is likely to improve the quality of education. Respondents were also asked to state (yes/no) whether they would endorse adoption of each principle by all parties involved in providing CME programs, including financial supporters, accrediting bodies, providers and faculty.
Overall, there were significant levels of endorsement of the principles by all stakeholders with 95%, 90% and 92% recommending adoption of principles 1, 2 and 3, respectively by all involved in provision of CME programs. A large majority (87%) also endorsed adoption of principle 4. Overall, responses were similar between types of organization.
Appropriate education
The Group adopted the following: CME providers should ensure that educational activities have clear learning objectives that are derived from a coherent and objective process that has identified performance gaps and unmet educational needs. The education must be designed to positively reinforce existing good practice and effect a sustained change in daily clinical practice as appropriate.
This principle was particularly well received with 94% of respondents agreeing or strongly agreeing that it sets an appropriate standard. A similar proportion (87%) agreed that the principle would lead to improved quality of education. Notably, 80% of respondents agreed or strongly agreed that implementation of the principle was feasible with 13% neither agreeing nor disagreeing. It was noted that accreditation guidelines from the EACCME require details for educational activities on how a needs assessment was performed together with a demonstration of how educational objectives were derived from that assessment. This requirement, combined with the aim of the Good CME Practice Group to provide a set of guiding principles that would set the highest standard of best practice, persuaded the Steering Committee to keep reference to a gap analysis in the principle.
Fair balance
The Group adopted the following: All relevant information should be disclosed to the learner so that they understand fully how the content has been developed and presented. This includes the terms of the financial support, relevant disclosures of faculty and organizations involved in the development of the scientific content, and the presentation of the program.
Response to this principle in relation to setting an appropriate standard stimulated a high level of endorsement of 90%. Similar high levels of agreement were achieved in relation to feasibility of implementation (90%) and likely impact on quality of education (93%). Only 3% either disagreed or strongly disagreed with any statement regarding this principle.
A challenge to this principle was received in the consultation phase highlighting that financial support for educational programs is not exclusively the domain of pharmaceutical companies but could be a medical society, academic institution or similar. In this situation, it is likely to be appropriate for the supporter to be involved in program and content development. The Steering Committee considered this feedback but felt that it is essential for the perception, credibility, integrity and success of CME program that content must be developed independently of all commercial organizations and their interests such that patient interests take precedence.
Transparency
The Group adopted the following: Balance needs to be evident in content, faculty and review. Content has to be developed independently of the sponsor and reflect the full clinical picture within the framework of the learning objectives.
Approximately two-thirds of respondents strongly agreed that this principle sets an appropriate standard and a further 31% agreed with this point. There was good agreement (82%) that this principle would lead to improved quality of education although 15% were undecided. Similarly, 79% agreed or strongly agreed that implementation of the principle was feasible.
Responses during the consultation phase and additional comments consistently reflected concerns about the feasibility of full transparency rather than its desirability. For example, more than one of the participating bodies suggested that the current system did not allow for sufficient transparency (for example, the tendency to show disclosures briefly on an opening slide) and that all organizations involved in an accredited activity should disclose their experience in educational programs. Based on these comments and the general feedback, the Steering Committee recommended that the drafted wording of the principle should be accepted.
Effectiveness
The Group adopted the following: Post-activity evaluation should measure satisfaction, knowledge uptake and intent to maintain or change behavior in line with learning objectives. To facilitate implementation of the core principles and ensure consistency of approach, the Good CME practice group have developed a checklist to guide providers through the fundamental processes in effective CME
A number of comments were received for this principle. A consistent point was that while acknowledging that a measure of effectiveness was necessary, this can be ‘difficult to implement’ as not all users completed an evaluation form. On reviewing the comments and data showing overall that there were doubts about the feasibility of implementation and likely impact on quality of education, the Steering Committee reworded the draft principle and removed the reference to the Moore scale.
Independence and transparency
At the time of inception of the Group, it was clear that there was a significant level of disquiet amongst most parties involved in provision of, and use of, CME in relation to
the quality and independence of programs. To a large extent, this reflects concerns around bias of programs due to industry support. But research from Germany and the US has indicated that industry-funded CME programs are statistically equivalent to non-commercially supported CME activities in the degree of detectable bias present14,15, underscoring the point that there are numerous potential sources of bias, that include intellectual and professional as well as financial biases. This point was recently acknowledged by the European Society of Cardiology in their CME policy document, which concluded that financial sources of bias must be acknowledged by full disclosure16. It is worth noting, nevertheless, that the relationship of perceived bias to actual bias is poorly understood as the difficulty of measuring actual bias has led to a paucity of studies in this area15,17.
Core principles around ‘fair balance’ and ‘transparency’ were developed to address the challenges of potential and perceived bias in educational activities and were most robustly supported and endorsed by those in the consultation phase. A key component of the ‘balance’ principle is that ‘content must be developed independently of the
sponsor’. Although this may be a self-evident means to avoid bias or undue influence, some accrediting bodies in Europe (particularly at national level) still allow significant input into content by the commercial supporter.
The Good CME Practice Group maintained that this should change and, where there is a lack of clarity in the regulations, this should be rectified by further discussion and engagement with the appropriate accrediting bodies. Moreover, the authors recommend that there is increased monitoring of programs and sanctions applied where necessary. While most people and organizations involved providing CME strive to produce high quality, objective and unbiased programs, ultimately, the user/learner should be in a position to assess this for themselves. They can only make informed decisions, however, if they are in full possession of information relating to potential conflicts of interest.
Scientific and medical journals have tended to be at the forefront of requiring comprehensive declarations of potential conflicts of interest and have defined conflicts of interest in relation to authors and financial supporters of publications. In a similar manner, both faculty and providers of CME-accredited programs should provide full disclosures of any personal conflicts of interest in addition to a clear declaration of source and terms of funding with details of their relationship, if any, with the funding organization. Learners should always be given the opportunity to report bias (perceived or real) through evaluation and feedback forms (provided at the point of learning).
While making learners aware of potential conflicts of interest is fundamental, it is also the responsibility of parties involved in the development of the content to address such conflicts as early as possible in the programme development. It is essential that content is evidence-based medicine (with only appropriate levels of evidence included), a balanced view of any therapeutic approach is presented and that any guidance given reflects consensus opinion from the medical community. All stakeholders involved in content development or presentation must agree and align with these concepts at the initial stages of the activity. If for any reason any party feels unable to do this, then consideration should be given to asking them to withdraw.
Appropriate and effective education
The goal of CME (and CPD) is to improve patient care through improved clinical performance of healthcare professionals translating into improved clinical outcomes. Evidence indicates that a major contributor to achieving this goal is a formal assessment of educational needs of the target audience. A requirement for a needs assessment is embodied in the first core principle of appropriate education and should give rise to learning objectives for the activity. The authors felt that it was important to crystallize the need to reiterate that course content encompasses current best practice and standard of care at this point.
There has been extensive debate over many years as to the how to measure educational outcomes and assess the effectiveness of CME and yet, the results of an extensive and systematic literature search reveal few publications on this subject. Undoubtedly, in many cases, assessment of some CME activities is more a measure of participation than learning or assessment of knowledge gain through correct completion of multiple choice questions (that are not always robustly designed). Ideally, since the goal of CME is to improve clinical care and patient outcomes, this should be the primary outcome measurement.
Haynes et al., reported a systematic analysis of 248 original articles relating to evaluation of CME. While 38% of studies measured change in clinical performance, only 7% of studies assessed the impact on patient outcomes. The authors note the challenge of measuring changes in patient outcome when this can be affected by many factors including patient-related delays in seeking medical care etc. This is particularly pertinent to this analysis since the studies reviewed were conducted predominantly in primary care.
A subsequent analysis of the literature relating to CME found that only 50/777 were randomised controlled trials that measured physician performance and/or health outcomes. Findings of this study highlighted the complexity of developing CME programs that can improve patient outcomes noting that enabling interventions were more effective that predisposing interventions. It was also noted that there was a lack of a robust association between change in physician performance and patient outcome.
This lack of clear relationship is likely to be due at least in part to confounding factors such as non-adherence to medications or even by therapy area (e.g., hypertension) where improvements are difficult to measure as there is often relatively good control and management. A recent study is more robust and has been able to show an association between learning programs in primary care and decreased mortality due to coronary heart disease3. This prospective, randomized, controlled trial assessed the effect of repeated case-based education of primary care practitioners on mortality. The interventions centred on regularly updating and distributing guidelines for lipid-lowering management combined with attendance at 1–2 case seminars/year over a 2-year period. At the end of the study period, mortality in the intervention group was 22% compared to 44% in the control group.
However, it remains difficult to get true measures of effective learning outside of a healthcare system for a number of reasons. Principal amongst these is the difficulty and expense of trying to measure change in behaviour or patient outcomes. Performance improvement CME has attempted to address measurement of change in clinical behaviour with a step in the assessment that assesses the user after an interval of time.
However this presents some challenges in terms of ensuring physician participation in the follow up particularly when there is no incentive for them to do. This is evident from the recent report from the Accreditation Council for Continuing Medical Education (ACCME) in the US26, showing that in 2011 only 0.75% of the total CME accredited programs (by number of hours) were classed as Performance Improvement CME.
A further challenge to measuring effective learning beyond the immediate MCQ assessment is that expectations of changing behaviour can be unrealistic when discussed in the context of a single event or 1-hour online course. Extensive consideration was given, therefore, to the final principle as regards how much guidance should be provided in terms of measurement of outcome. It was thought that at this time, that not only is it difficult to measure change in patient care but the Group felt that a physician’s manager or peer group were best placed to review a physician’s progress following CME activities. The Group would recommend that participation in, and learning from, such CME activities become a component of the formal review system in all countries. The first draft of the principle included reference to the Moore scale as an objective measure of successful learning
However, in the consultation process, this proved to be the most contentious principle with comment that this was not feasible for many and that there was a trend towards assessment of level of satisfaction as an adequate/ realistic measure. The Steering Committee felt, however, that while it would remove reference to the Moore scale, it was important to recommend that, in addition to assessment of user satisfaction and knowledge uptake, a measure of intention to maintain or change behaviour was essential and should be included as such in all evaluations. Further research on novel, practical and affordable ways to assess change in practice may help to identify reasonable methods for achieving more meaningful evaluations and driving improvements in education standards.
Conclusion
In summary, the overall goal of the Good CME Practice group is to provide guidance on how European CME providers can contribute to improving clinical care. The group aims to do this by championing best practice in CME, maintaining and improving standards, mentoring and educating and working in collaboration with critical stakeholders. By this means, CME programs should be better targeted to address true educational needs of learners. Moreover, improved design of initiatives will lead to effective education that translates to positive changes in the clinical practice of users.
Key to success of this is to ensure that the core principles developed in this initiative relating to (a) appropriate education, (b) effective education, (c) fair balance and (d) transparency are recognised and implemented by all involved in the CME arena along with continued research efforts in the impact of these principles on practice.