The Food and Drug Administration (FDA) recently published an Issue Paper discussing the use of surveys to assess whether a risk evaluation and mitigation strategy (REMS) is meeting its goals related to knowledge of health care providers and patients.
FDA often requires a REMS to include a goal to inform or educate patients or health care providers (prescribers and/or pharmacists) about the risks associated with a drug. The important messages may include information about how to use the drug safely. REMS assessment surveys are one way to assess the extent to which patients/caregivers or health care providers understand the risks associated with the drug and/or how to use it.
Since the initiation of required REMS assessments four years ago, FDA has reviewed numerous REMS assessments that contain survey data and has identified issues associated with the methodologic rigor of the surveys, calling to question their utility as a tool to assess whether REMS are achieving their goals. Accordingly, FDA’s issue paper focuses on the use of surveys to assess patient and provider knowledge.
Background
Title IX, Subtitle A, section 901 of the Food and Drug Administration Amendments Act (FDAAA) (Public Law 110-85)2 created section 505-13 of the Federal Food, Drug, and Cosmetic Act (the FD&C Act). Section 505-1 authorizes FDA to require applicants submitting NDAs, ANDAs, or BLAs to submit and implement a REMS if the FDA determines that a REMS is necessary to ensure that the benefits of a drug outweigh the risks of the drug. To require a REMS for an already approved drug, FDA must have become aware of new safety information as defined in the statute.
The majority of REMS include, as a goal, informing or educating patients or health care providers of the serious risks of a drug. Thus, the majority of the current REMS have one or more educational components, such as a Medication Guide, a communication plan, or a communication-related Element to Assure Safe Use (ETASU).
FDAAA requires NDA and BLA applicants to periodically assess the effectiveness of their product’s REMS. REMS assessments generally include an evaluation of the extent to which each of the REMS elements are meeting the goals and objectives of the REMS, and whether or not the goals, objectives, or REMS elements should be modified.
For REMS containing these educational components, the assessment plan generally requires an assessment of patient or health care provider understanding of the risks of the drug and/or safe use of the drug. The first REMS assessment was submitted on September 27, 2008.
As of December 31, 2011, FDA has received and reviewed 144 assessments that contained surveys for 105 individual drugs (many of these assessment submissions, particularly for those associated with REMS with ETASU, represent serial assessments (e.g.., assessments every 6 months)). A total of 55 of these REMS were of the Medication Guide-only type; 33 had a Medication Guide and a Communication Plan; and 56 were classified other, which were predominantly REMS with ETASU.
FDA requested and applicants submitted only cross-sectional surveys to evaluate patients’ or health care providers’ understanding of the risks of the drug and/or safe use of the drug. In addition, in a July 2010 public meeting on REMS, the public speakers who addressed this issue noted that surveys are the accepted methodology for evaluating understanding and knowledge. It is also important to note that the assessment of knowledge is not the entire scope of a REMS assessment; REMS assessments may contain data from sources other than surveys to inform the overall effectiveness of a program.
FDA Practices Regarding Survey Methodologies
To date, FDA has generally requested that applicants submit their proposed survey methodologies to FDA for comment at least 90 days before fielding the survey. FDA adheres to general social science principles in advising applicants. Most FDA comments on the survey proposals have related to the methods proposed and the wording of individual questions. High-level concepts that FDA communicates to applicants are summarized below:
- State the purpose of the survey and how it relates to the REMS goal(s)
- Identify the key risk messages that should be understood by the group of interest (patients, prescribers, or dispensers), depending on the goal of the REMS
- Enroll a demographically diverse population that has used, prescribed, and/or dispensed the product, depending on the goal of the REMS
- Specify a survey design, including a statistical analysis plan, that meets the survey objectives, and calculate the appropriate sample size
- Construct a questionnaire that targets the key risk messages
- Minimize factors that might contribute to a biased survey (e.g., unrepresentative sampling, faulty recruitment strategies, leading questions that bias the responses in a particular direction, missing data)
- Minimize any burden imposed on participants taking the survey
FDA’s Experience With Knowledge Assessment Surveys
Surveys conducted as part of a REMS assessment have focused on patients’ or health care providers’ understanding of the key risks of the drug. FDA does not expect those surveyed to recall the full prescribing information or all of the information available for patients. For example, patient surveys are generally limited to the information included in the “Most Important Information” section of the Medication Guide. In rare instances, prescriber surveys have included questions about prescribing behavior.
A high-level summary of FDA’s experience with surveys to assess REMS follows (through December 31, 2011):
Selection criteria: Applicants have almost exclusively used convenience samples (generally defined as when respondents are selected based on the basis of what is most feasible for the researcher). Although applicants sometimes send invitations to potential participants in a random manner, the actual survey populations generally do not represent random samples. As described in Recruitment strategies below, generally, invitations are sent to adult, English-speaking patients who have been dispensed the drug or to prescribers who have prescribed the drug in the previous 3 to 6 months. Responding to the invitation is voluntary, and it is possible that respondents may attend to health or prescribing information to a greater extent compared to the entire population on the drug, or respondents may be more interested in the incentives offered for completing the survey.
Recruitment strategies: Applicants have most commonly used the following techniques to recruit health care providers (not all inclusive):
- Sending letters by mail, fax, or email or placing a telephone call to known or potential prescribers identified by pharmacies or national audit services
- Sending a fax or other communication to a random sample of relevant potential prescribers identified by specialty (e.g., rheumatologists for a rheumatology product) or general likelihood that an invitee may have prescribed a drug (e.g., a random sample of internists for a diabetes drug)
- Contacting prescribers using the prescriber registry (where applicable such as when prescriber certification is required by the REMS)
Applicants have most commonly used the following techniques to recruit patients (not all inclusive):
- Using the pharmacy or pharmacy network to identify patients dispensed drug and invite them to participate (usually by mail)
- Having the pharmacy recruit patients personally while presenting and dispensing the prescription
- Requesting that prescribers identify eligible patients and provide an invitation. Alternatively, for drugs administered in an infusion center, by soliciting all or a subset of infusion sites (when the patient presents for dosing)
- Requesting that prescribers personally recruit patients at the time of prescription writing
- Contacting patients using the patient registry (where applicable such as when patient enrollment is required by the REMS)
- Providing an invitation with materials that were supposed to have been dispensed with the prescription
Response rates: It is not uncommon for applicants to have a low response to their survey recruitment efforts; it is not uncommon to see recruitment rates ((number completing the survey/number invited) X 100) below 10 percent.
Sample size: For health care providers, actual sample sizes have ranged from 3 to approximately 500. For patients, the sample size has ranged from 8 to approximately 400. For surveys with a low sample size, this is usually attributed to small patient populations.
Question types: Questions are generally limited to the True/False/I don’t know or multiple-choice (either the single best or all applicable answers) types. We have seen few open-ended questions or questions related to a case presentation.
Statistical Analysis: Some applicants have prespecified a threshold for success. Formal statistical hypothesis testing is generally not conducted. Applicants compile and report descriptive statistics summarizing the demographics of the surveyed population and the proportion of subjects who demonstrated knowledge of the risk message (defined by us as the knowledge rate). Rarely do applicants compare the demographics of the surveyed population to the population that uses, prescribes, or dispenses the drug to determine if the sample is representative.
Findings: Generally, the surveys submitted as part of a REMS assessment have shown:
Patient surveys: There is evidence of limited understanding of some important risk information (in some surveys, patients had < 50% correct answers to questions about critical risks). A small FDA analysis suggests limited survey representativeness.4 For the limited number of surveys that stratify results by whether or not the patient reported reading the Medication Guide, readers tended to have higher scores. The significance of this finding is not clear.
Health care provider surveys (predominantly prescribers): Generally, surveys show that the REMS are meeting their information goals (as defined by a knowledge rate ≥80%). In some cases, adequate performance on all but one of the risk questions has resulted in modifications to the educational materials.
In the context of the goal of a valid, reliable, and useful knowledge/understanding assessment, our review of the REMS survey data has identified issues regarding the adequacy of some of the surveys conducted to date:
- Because of the limitations outlined below, surveys may not be the best methodology in all situations to assess knowledge in populations that range in size from hundreds to potentially millions.
- Sometimes the sample size achieved appears to be too low to draw conclusions with confidence.
- The surveyed population may not reflect the demographics of the target patient population.
- Because convenience samples are commonly used, surveys are likely to be biased in some way.
- The level of difficulty of the questions varies across surveys, although this is a subjective assessment that has not been formally assessed. This variability may be related to:
- Familiarity of the information to be conveyed (e.g., the risks of a first-in-class drug may be less familiar than those of a drug that has been marketed for a substantial duration of time).
- Wording of a question and the level of detail of knowledge tested. In some cases, the correct answer is self-evident.
- Sometimes the wording of a question is open to interpretation, which may explain a low knowledge rate.
- There are no objective standards for an adequate knowledge rate (i.e., the threshold for success) of the REMS educational component.
- There may be alternatives to the knowledge rate to assess the effectiveness of the educational component of the REMS.
Input Requested
Most REMS require drug manufacturers to provide educational materials for health care providers, patients, or both, and FDA is seeking input on the best way to evaluate the effectiveness and utility of these requirements.
Ideally, REMS assessments would provide information about whether health care providers prescribing or dispensing the drug and/or patients taking the drug understand the key risks and safe use of the drug (where applicable). Having a valid, reliable, and accurate measurement of the level of understanding would answer two critical questions related to the informational goal: Is this component of the REMS working? And if not, how should the REMS be modified? In addition, as FDA has gained experience with REMS, the following questions have taken greater prominence:
- Is the REMS affecting health care provider and patient behavior?
- What are the effects of REMS on burden to the health care system?
- Is the REMS adversely affecting patient access to the drug?
Accordingly, FDA is seeking public input on the following questions:
- What strategies can the applicant use to recruit a sample that is representative of the population that is prescribing/dispensing/taking the drug?
Given that the applicant cannot compel an individual to complete a survey, is it acceptable to enroll a relatively small (making the survey feasible) number of participants that are representative of the totality of the health care provider or patient population and make generalizations from that sample to the larger population?
What is an adequate sample size to be able to confidently extrapolate findings to the entire population prescribing/dispensing/taking drug?
2. Is the knowledge rate (i.e., the proportion of subjects who demonstrate knowledge of the risk message) the appropriate primary endpoint for a survey?
What factors need to be considered when establishing the threshold for success for educational elements of the REMS?
Should the threshold for successfully meeting a REMS educational goal be set at a knowledge rate of 80 percent or 90 percent, or should it vary depending on the risk message? If it should vary, what should the minimum threshold for success be? Should the threshold reflect whether the product is a new molecular entity (NME) or original biologic product, or an older drug?
3. Since most surveys use only True/False and multiple-choice questions, what are the advantages and disadvantages of using other question types (open-ended, case vignettes, fill-in-blank) to evaluate knowledge?
4. Please discuss process issues related to these surveys:
-
- Given issues of recall, should the lag time between the REMS communication and the survey administration be standardized?
- Should pretesting/validation be required to reduce the likelihood of a poorly worded question that was not recognized during survey development?
- On average, how long does it take to design, test, recruit participants, conduct, analyze, and report the results of a survey?
- Please comment on appropriate incentives for patients and health care providers to complete surveys.
5. Given the issues with surveys that FDA has observed, what are the alternatives to knowledge surveys to assess the effectiveness of the educational elements of the REMS? If any, what are the advantages and disadvantages of the alternatives?
6. What are the considerations in designing questions to assess the impact of REMS on patient and/or provider behavior and access to drug, as well as the potential burden of the REMS on these groups? What are alternative methods to assess behavior, burden, and access for a REMS? If any, what are the advantages and disadvantages of the alternatives?
7. Question for industry: From your perspective, what challenges have you encountered in designing and conducting knowledge surveys?