In recent years, significant data integrity lapses have occurred at global pharmaceutical, biotechnology, and medical device firms. Repercussions have included new products not being approved, rescinded approval of products on the market, closure of manufacturing plants, massive product recalls, FDA import alerts, and drug shortages.
At the same time, data is proving crucial to advancing medical products that cure or prevent disease. Clean, solid data also underpins successful machine learning projects within life science industries. Artificial intelligence applications hold significant promise for developing new patient therapies.
Yet the medical device and pharmaceutical industries remain hesitant to expand their use of artificial intelligence. Over the past century, healthcare-related industries have amassed immense amounts of data that remain untapped. This data could potentially be used for unprecedented and groundbreaking analyses. The integrity of the data must be assured; otherwise, those efforts will be wasted.
Fortunately, for those who want to ensure the integrity of their data, there are tried and true methods for performing that activity. Recent developments even provide a more modern, holistic look at data integrity.
At the Xavier Artificial Intelligence (AI) Summit held virtually in August 2020, a session addressed data integrity and the challenges for artificial intelligence. Industry veteran Steve Niedelman discussed the importance and criticality of data, the consequences of data integrity failures, and how to find and eliminate data integrity issues.
Niedelman is the Lead Quality System and Compliance Consultant in the FDA and Life Science Practice at Washington, D.C. office of law firm King and Spalding. He completed a 34-year distinguished career with FDA and serves as a consultant to medical device, drug, biologics, tobacco, and other FDA-regulated industries. Although not an attorney, Niedelman is an FDA drug, biologics, and medical device expert.
“Solid data integrity provides a foundation for the quality and safety of pharmaceuticals and medical devices in the product development and manufacturing phases of the product lifecycle,” Niedelman maintained. “It plays a fundamental role in ensuring that the decisions being made by a company have the supportive documentation behind them and that they can be defended.”
FDA: Under a Data Integrity Microscope
“It is important to understand that everything you do, everything FDA will look at is based upon the integrity of your data,” Niedelman emphasized.
Any discussion about data integrity requires first understanding the language FDA uses internally to define high-quality data.
How does FDA define high-quality data? Their definition: “Data are of high quality if they are fit for their intended uses in operations, decision-making, and planning . . . as data volume increases, the question of internal consistency within data becomes paramount….”
It is “extremely important” to understand this, he said, “and it is followed in everything that you do in support of development of data for clinical work, clinical data, clinical studies, machine learning, and artificial intelligence. You need to be able to stand behind the data that you are creating and the data that you are using to make your decisions. That will include the importance of following data integrity principles.”
“Today, I will be presenting information on a mnemonic that is currently in place that is used by many of the FDA regulated industries,” Niedelman said.
Solid data integrity provides a foundation for the quality and safety of pharmaceuticals and medical device
The mnemonic was first used by Stan Woollen of FDA’s Office of Enforcement in the 1990s, and during subsequent years it was expanded with input from EMA. Woollen coined “ALCOA” as a memory device to ensure data is Attributable, Legible, Contemporaneous, Original, and Accurate. Subsequently, EMA input broadened it with additional terms to “ALCOA Plus.”
“It is important that you can support the data, and it is important that you basically follow the premise of ALCOA, which has now grown to become ALCOA Plus. It has expanded over time based upon experience in the industry and learning that there were additional elements to consider in this mnemonic that was created to assure accurate and timely and contemporaneous data development.”
Consequences of Data Integrity Failures
Failure to ensure data integrity, Niedelman pointed out, may result in undesirable results including bias, concept drift, and ineffectiveness, all of which can affect future decisions in product development—for example, whether to go forward with a product in development. In addition, it may result in unvalidated outcomes that cannot be supported, only to result in post-market issues and problems down the road, including product recalls.
It may also affect a firm’s reputation in the eyes of regulators. If data integrity issues are identified during inspection or product reviews, that could result in the firm receiving a warning letter or an untitled letter. Data integrity remediation has been included in several negotiated consent decrees with firms that have been subject to injunction.
It is essential that data integrity serves as the fundamental underpinning to assuring that what your firm is doing can be substantiated. In other words, you must be able to stand behind it and have consistent controls over the data.
Routine data integrity issues left unchecked can lead to frequent product recalls. Firms that routinely recall product impact customer loyalty when the customer sees a pattern of continuing problems.
“It is essential that you control your data from the outset,” Niedelman stressed. “Make sure you have consistent oversight, even for data that is being collected in other geographic areas where oversight may be lax. Knowing that in advance, it is your responsibility to make sure that appropriate oversight is in place, procedures are being followed, and audit trails are in place. Have 21 CFR Part 11 compliance systems in place and ensure that employees have been trained and are following those requirements.”
Too frequently employees have not been appropriately trained on Part 11. Many of the data integrity problems that FDA has identified are associated with nonconformance to 21 CFR Part 11, where people are using shared passwords or people are signing for outputs when they were not even at work that day.
“It just indicates that somebody else is using those passwords, and it raises the question as to who is date stamping that if that person was on leave that day,” Niedelman pointed out. “Those are the kinds of problems that in part come from ineffective employee training. But it also emphasizes the importance of understanding Part 11 to assure that your data integrity is protected.”
Routine data integrity issues left unchecked can lead to frequent product recalls
The Importance and Criticality of Data
“Criticality of data serves as the underpinning of everything from concept, to feasibility studies, to pre-market, to manufacturing, to post-market, and unfortunately to recalls and subsequent regulatory actions that may be necessary,” Niedelman pointed out (Figure 1).
Figure 1 Importance and Criticality of Data
“It is all based upon the integrity of the data that you have in hand and are using to make decisions. It is extremely important to ensure the integrity of the data and to be able to stand behind it and understand it and defend it.”
“When regulators perform an inspection or review drug or device applications, they need to be able to rely on the data they are given. When they see patterns of data that are too perfect, it raises questions,” Nidelman explained. “Similarly, they can pull apart data and identify some of the problems I just mentioned associated with the data integrity and employee practices that raise questions about the data that is being reviewed.”
The firm should be able to provide as many assurances of product safety, strength, purity, quality, as possible, all of which depend on the data being discussed. Any data-driven decisions must be reproducible and based upon data that can support those outcomes. Personnel need to be trained to understand the importance of Part 11 requirements and of following them.
“While integrity of data is not a new issue, it is one of increasing attention at FDA,” Neidelman said. “Investigators are being trained. They are receiving specialized training to detect data integrity issues, data manipulation, and in the worst case, when it may be intentional and fraudulent, which then could become criminal in nature.”
As a result, it is important that you maintain vigilance over your data from the very outset. You do not want to erode your public confidence or impugn your product quality. That can have an impact on your bottom line and an impact on your firm’s integrity going forward.
When they see patterns of data that are too perfect, it raises questions
Some of the issues are not always straightforward. It could be the geographical areas ranging from domestic to international, depending on level of oversight. Different countries do not understand the same importance that FDA understands regarding the level of oversight that may be expected over the data. It does not matter whether you are a large firm or small firm. It could be an individual employee, or there may be a conspiracy to hide or share data, skew data, or destroy data, which could be viewed as criminal.
Many times, FDA will walk into a firm—generally a foreign firm—and find a whole bunch of data being shredded or already dumped in a trashcan. That raises issues as to the actual integrity of the data. After all, why isn’t it being stored? How is it able to be shredded so easily? All of that contributes to the integrity of your data.
“You need to understand the data that is generated both pre-market and post-market can be critical to your firm,” Niedelman said. “Acts of commission, as well as omission, are equally important. If you eliminate data unnecessarily or hide data or do not report all the results, these all raise data integrity issues. FDA has been on top of data integrity for a while. While it is certainly more prevalent in pharmaceuticals right now, it is growing in the medical device area as well.”
ALCOA Expands
Returning to ALCOA, over time additional elements were added to the mnemonic and it became ALCOA Plus to assure that that data was also complete, consistent, enduring, and available (Figure 2).
Figure 2 ALCOA Plus
Niedelman commented on what each of the various elements means for data:
- Attributable: Specific to staff members via audit trails and e-signatures. No sharing of passwords. Requires compliance to Part 11.
- Legible: Usable and readable during internal audits
- Contemporaneous: Recorded during performance of a task, not before or later
- Original: In the original format, not manipulated, stored elsewhere, or shared
- Accurate: Maintained using a system that minimizes errors, raw data and analytical results correctly presented
The additional elements include ensuring that the data is:
- Complete: Everything is included and nothing is missing
- Consistent: Enforcing the use of approved data acquisition and analysis methods
- Enduring: Use of a media that maintains and protects records and it does not deteriorate over time
- Available: Records available as needed
And, it is “really important to understand that following the basic tenets of ALCOA Plus provides you the assurance that your data as building blocks will get you to the endpoint where your data will be useful in making sound decisions.”