In September, FDA issued a draft guidance to clarify how it evaluates real-world data to determine whether it can be used in FDA regulatory decision-making for medical devices. The guidance explains how FDA will use data collected outside of traditional clinical trials, such as electronic health records and registries, as a part of the medical device process. This data, as the guidance explains, is only appropriate if it represents the population being studied.
Introduction to Real-World Data (RWD) and Real-World Evidence (RWE)
The guidance outlines the FDA’s definition of these terms, considering real-world data (RWD) to include that that is collected from sources outside of traditional clinical trials. These sources may include large simple trials, or pragmatic clinical trials, prospective observational or registry studies, retrospective database studies, case reports, administrative and healthcare claims, electronic health records, data obtained as part of a public health investigation or routine public health surveillance, and registries (e.g., device, procedural, or disease registries). The data is typically derived from electronic systems used in health care delivery, data contained within medical devices, and/or in tracking patient experience during care, including in home-use settings. Furthermore, real-world evidence (RWE) to be evidence that is derived from the aggregation and analysis of RWD elements.
FDA explains that routine clinical practice often involves the use of cleared or approved devices for uses or in patient populations not within the cleared or approved indications for use. However, the advances in knowledge that may result are often not realized because the data collected are not systematically characterized, aggregated, and analyzed in a way such that it can be relied upon to inform regulatory decision-making. By recognizing the value of RWE as an important contributing factor for understanding and regulating medical devices, FDA wants to encourage medical device researchers, manufacturers, physicians, hospitals and other stakeholders to learn more from routine clinical care.
FDA will use the criteria described in the guidance to help determine if RWD data sources are of sufficient quality to potentially generate valid scientific evidence FDA relies only upon valid scientific evidence to determine whether there is a reasonable assurance that a device is and effective. While it is required that this bar be met in all such cases, it is possible that RWD could meet this threshold under circumstances when important and necessary patient data were accurately and reliably captured at clinically relevant time intervals throughout the appropriate portions of the lifecycle of the medical device.
For example, RWE may be suitable to support the expansion of the indications for use of cleared or approved devices through an appropriate premarket submission. RWE may also be suitable to augment the information needed to support clearance or approval of the next generation of a device. Other applications of RWE in premarket decision-making may be possible, as well, particularly as data systems and analysis methodology advance.
Considerations of RWE
FDA states it will consider the use of RWE to support regulatory decision-making for medical devices when it concludes that the clinical data contained within RWD source(s) used to generate the RWE are of sufficient quality to provide confidence in the analyses necessary to inform or support the regulatory decision throughout the total product life cycle. The threshold for sufficient quality will depend on the specific regulatory use of the evidence.
For example, a specific patient registry might be informative for postmarket surveillance, but not adequate for a premarket determination of safety and effectiveness, while another patient registry may be suitable to address both pre- and postmarket evidence requirements.
The guidance notes that RWE may potentially be used in many ways to understand medical device performance at different points in the total product life cycle, offering several examples, such as:
• generation of hypotheses to be tested in a prospective clinical study
• as a historical control, a prior in a Bayesian trial, or as one source of data in a hierarchical model or a hybrid data synthesis
• in a setting where a registry or some other systematic data collection mechanism exists, RWD can potentially be used as a concurrent control group or as a mechanism for collecting data related to a clinical study to support device approval or clearance
• in some circumstances where real-world use of a device is in a broader patient population or wider set of circumstances than described in the device labeling, it may be possible to use existing systematically collected RWD to expand the labeling to include additional indications for use or to update the labeling to include the new information on safety and effectiveness
Characteristics of RWD
FDA states it does not endorse one type of RWD over another. RWD sources should be selected based on the ability to address specific regulatory questions. Collection of RWD should not dictate, interfere with or alter the normal clinical care of the patient, including choice of treatment.
Important factors regarding RWD that FDA will assess include the relevance and reliability of the source and its specific elements. The underlying data should be robust (i.e., provide meaningful information under a variety of conditions) for the purposes and analyses for which it was designed. These assessments will be used to determine whether the data source(s) and the proposed analysis generate evidence that is sufficiently robust to be used for a given regulator purpose. That is, the threshold for whether RWD is sufficiently relevant and reliable for use will depend on the level of quality required and/or necessary to make a particular regulatory decision. These factors for assessing the value of RWD sources apply to all FDA regulatory uses of the data.
Examples
FDA outlines several examples it describes as “generalized” from actual regulatory uses of RWE for regulatory decision making:
Expanded indications for use
The National Cardiovascular Data Registry (NCDR) was created in 1997 by the American College of Cardiology (ACC) as “an exploration into strategies for improving cardiovascular care through the use and application of clinical data.” These registries are designed to help participants measure, benchmark, and improve cardiovascular care. In particular, the Registry for diagnostic cardiac CATHeterization and Percutaneous Coronary Intervention “assesses the characteristics, treatments and outcomes of cardiac disease patients who receive diagnostic catheterization and/or percutaneous coronary intervention (PCI) procedures, measuring adherence to ACC/AHA clinical practice guideline recommendations, procedure performance standards and appropriate use criteria for coronary revascularization.”
As a registry collecting data on consecutive patients and focused on quality assessment/performance improvement data related to real-world procedures and device use outcomes, an IDE is not required for routine data collection operations, even though a substantial volume of data is generated from use of a device, including data on use outside of the cleared or approved indications for use.
Postmarket Surveillance Studies
FDA has issued a series of postmarket surveillance study orders, related to investigating patient safety issues in a type of class II device, under the authority of Section 522 of the Federal Food, Drug, and Cosmetic Act. These 522 orders covered multiple devices from different manufacturers that are similar in intended use, design, and other characteristics, such that the surveillance questions were identical. To comply with the orders, many manufacturers decided to collaborate with a clinical professional society in this field and with FDA to develop a patient registry that would collect needed data to address the public health questions. The resultant registry was designed to collect data on all patients with the condition, including those treated with the devices of interest, other devices, and through medical management, and to follow their treatment outcomes
Post-Approval Device Surveillance as Condition of Approval
Permanent implants are typically designed to serve patients for a time period that is much longer than what can reasonably be captured in a premarket clinical trial. For example, a trial that follows patients for two years after implantation would not produce data for the designed life span of 7 to 10 years for that implanted device. Traditionally, FDA would require extended follow-up of the premarket patient cohort and an additional new-enrollment study designed to capture hundreds to thousands of patients with follow-up for the life of the implanted device.
Some clinical professional societies have developed registries that collect data on patients receiving these devices. FDA has worked with manufacturers and professional societies to evaluate the registries and has found that they can be reliable for certain health outcomes of interest. Should a registry exist that is capable of addressing the questions for which a Post-Approval Study (PAS) may be issued, FDA instead may issue a condition of approval that a manufacturer participate in and support collection/reporting of registry data on their device in lieu of a condition of approval specifying a formal PAS.
Control Group
A manufacturer approached FDA during the development of a new medical device that had substantial technological changes from previous iterations of that specific device and other similar devices from other manufacturers. FDA determined that additional clinical evidence was needed to support an approval decision for this device. A registry exists that captures all uses of medical devices in this clinical indication. The manufacturer designed a clinical study that compared the use of the new device to a non-randomized concurrent control group derived from the registry. The existing registry was evaluated by FDA and the manufacturer according to the factors cited in this guidance and was found to provide sufficient data on the control population, such that the manufacturer did not have to collect additional data from these patients or influence the course of their clinical care in any way.
Supplementary Data
FDA evaluates available evidence to make the best decision for patients and public health. In the case where RWD has been systematically collected, FDA has used these data, in combination with case reports, publications, adverse event reports, engineering and nonclinical test data, and other sources of information available to FDA to provide a full understanding of the severity of the issue, precipitating factors, affected population and alternative therapies. Periodically, FDA identifies an issue related to the safety of a marketed medical device that was not detected in premarket trials. The addition of RWD has proven extremely valuable to FDA, patients, physicians, and manufacturers to develop a course of action that best protects public health in these instances.
Objective Performance Criteria and Performance Goals
An Objective Performance Criterion (OPC) refers to a numerical target value derived from historical data from clinical studies and/or registries and may be used in a dichotomous (pass/fail) manner by FDA for the review and comparison of safety or effectiveness endpoints. An OPC is usually developed when device technology has sufficiently matured and can be based on publicly available information or on information pooled from all available studies on a particular kind of device.
Similar to OPC, a performance goal (PG) refers to a numerical value that is considered sufficient by FDA for use in the evaluation of an investigational device regarding a safety and/or effectiveness endpoint. But, generally, the device technology is not as well-developed or mature for use of a PG as for an OPC, and the data used to generate a PG is not considered as robust as that used to develop an OPC. A PG might be considered for challenging patient populations or if there is no clinical equipoise for any control. From a sufficiently relevant and reliable observational data source, a PG can be constructed using appropriate statistical methods, such as a subject-level meta-analysis. As technology evolves over time, an OPC or PG could be updated using observational data.