An FDA drug investigator is reporting that a lack of consistency in the interpretation of standard operating procedures (SOPs), in terminology, and in how various types of data are stored, tracked, and trended in drug companies is introducing unneeded risk into their quality management systems.
At a PDA Midwest Chapter meeting in Indianapolis in early October, FDA Office of Pharmaceutical Quality Operations (OPQO) Consumer Safety Officer Emilie Kahn shared her take on the increased risk that companies are unknowingly introducing into their quality risk management (QRM) systems.
This is such a simple idea, but I have seen it in every firm that I have been in, whether they are violative or not
“This is such a simple idea, but I have seen it in every firm that I have been in, whether they are violative or not,” Kahn said. “Different people are interpreting SOPs differently. They are using 30 or 40 different synonyms for the same issue. And how the firm is housing, tracking, and trending that data introduces all kinds of variation into their risk assessment that is not necessary.”
Kahn’s duty station is in Detroit, Michigan. She is in OPQO Division 3, which now includes the previously geographic-based districts of Chicago, Cincinnati, Detroit, Kansas City, and Minneapolis.
[NOTE: Are you looking for enforcement records that cite a specific keyword? Get a free 483 Observation Report that analyzes 25,000+ 483s for the FDA issue you’re looking for.]
Explaining the QRM Framework
Kahn told attendees that she uses QRM principles when inspecting quality units. “I am using the same principles that you are using in your firm when I evaluate your firm,” she explained.
Kahn pointed to the 2006 FDA guidance, “Quality Systems Approach to Pharmaceutical CGMP Regulations” and read two key paragraphs from the guidance that she wanted to focus on in her presentation:
“The quality system provides the foundation for the manufacturing systems that are linked and function within it. The quality system model described in this guidance does not consider the five manufacturing systems as discrete entities, but instead integrates them into appropriate sections of the model.”
“The FDA has concluded that modern quality systems, when coupled with manufacturing process and product knowledge and the use of effective risk management practices, can handle many types of changes to facilities, equipment, and processes.”
“So, we are looking for manufacturers with robust quality systems and appropriate process knowledge and evaluating how they implement improvements,” Kahn said.
She commented that the Quality system is always inspected and scrutinized because it is “the necessary framework for implementing quality by design and continual improvement using quality risk management. Each company should tailor it to their specific environment.”
By drawing on the Q9 guidance, Kahn provided her view of the key qualities of effective risk management. “It is systemic, it is structured, and it facilitates science-based decision making regarding drug product quality,” she maintained.
The consumer safety officer pointed to some key themes from ICH Q9 (see Figure 1).
Figure 1
“A robust system can control variability and it can remain effective,” Kahn commented. “I interpret this to mean that the data is constructed to survive and function in multiple settings. It is reusable, consistent, can be updated, and it is reliable.”
Using QRM Principles to Inspect QRM
Kahn explained the general approaches investigators take in performing drug GMP inspections and how her approach includes a QRM element.
She starts in advance of the inspection with some research on the firm being inspected, looking at several things, including the age of the site, inspectional history, and timing of the inspections—paying attention to previous observations, including what areas, systems, and products they were in. Areas, systems, or products where observations were found in the past will be looked at again.
After arriving at the plant, she requests a lengthy list of documents, including:
- Complaints
- Rejections
- Deviations
- Returned goods records
- Current staff competencies
- Raw material data
- OOS results
- Results of audits and inspections
- Maintenance issues
- Stability results
- Process trending data
The requests are made in part to evaluate how a firm is collecting, housing, trending and using those various kinds of data. “These documents are not system-specific,” Kahn commented. “But you will see that they cover all of the six quality systems.”
Following the document requests, Kahn performs a walk-through of the manufacturing facility and prioritizes which products to focus on based on any concerns she sees in the plant—products for which she will request additional information including deviation reports, corrective and preventive action (CAPA) plans, and out-of-specification (OOS) results. She then follows that information focusing on QRM fashion on issues that have the most potential risk to patient safety and public health.
“During the course of the inspection, we are looking to see how the quality unit is utilizing this information and whether small problems are snowballing into larger problems,” she explained.
“Does management only become aware of issues after a crisis? Is management reacting? Or are they correcting problems and taking responsibility? Are they patchwork corrections or are systemic corrections in place? The categories of documents I mentioned together elicit answers to those questions.”
Kahn noted that an FDA investigator operates in a “gray area” in which he or she does not have all of the information, similar to what a quality unit may be doing—“especially if a problem only presents itself way downstream and the quality unit does not have all the information it needs.”
Regarding deviation investigations, Kahn emphasized that the principles of QRM are of special importance.
“I ask for the status of each open deviation investigation and look at whether the level of investigation is commensurate with the level of risk that product may pose to the patient. I look at which avenues are pursued in the investigations—which are evaluated, and which are not,” she said.
Key Takeaways Focus on Data Reliability and Variability Reduction
In the “key takeaways” section of her presentation, Kahn again pointed to the important aspects of effective risk management and how the quality of the data impacts QRM outcomes.
Noting that effective QRMs are systemic and structured, science-based, and use a robust set of data, she focused on the data quality aspect.
“Why is the quality of the data underlying all of the document requests so important?” she asked. “Because it is garbage in garbage out. If you have terrible data that is inconsistent you are not going to be able to get any reliable outcomes from it.”
She explained that for “any data that you are collecting on any problem with any system you are taking qualitative narrative for an issue that happened and you are translating that into quantitative data that you are using to provide a risk assessment. The question is, are your methods of documentation and ultimately the quality of the data that you are collecting from qualitative narrative issues introducing additional risk into your management system?”
Kahn stressed the importance of consistently evaluating specific types of issues in specific ways. “Whether it is CAPAs or OOSs or deviations, data needs to be collected in a reliable and consistent manner. The more staffing and departments you have involved in the investigative processes the more opportunities there are for variance in interpreting SOPs. A key aspect for me is whether the quality unit is recognizing this potential variation.”
If I am on-site and I am trying to track and trend an issue that I see, and they can’t even provide me with the raw data to answer that question—if I can’t do it, they can’t do it. Which means they haven’t been doing it
During inspections, Kahn and other investigators evaluate data quality by talking with plant personnel. “We interview people. We assess whether there are aspects of their procedures that are open to interpretation and how the staff may interpret those procedures differently.”
In addition, she said, “We do an assessment of the quality of the information that you have relative to its purpose and its ability to serve that purpose. [We look for] data quality metrics that are complete and accurate—whether the SOPs are easy to follow, whether the terms in the SOPs are defined. A lot of issues only come out during personnel interviews when the filter might be more of making blame in the organization. That actually reveals a lot in some of these deficiencies with data quality.”
Kahn said that she has seen instances—for example, in production deviation investigations—where during a search for historical tracking and trending, terms being searched on were inconsistent or the searches were not broad enough.
“They are almost willfully putting blinders on. For example, when they run Trackwise they are only looking for a couple of terms related to the root cause. But the root cause could be affecting a line or a piece of equipment or a person. A key aspect for the quality of the data and whether it is consistent, reliable, and effective starts with really basic data collection practices, such as defining terms and defining scope.”
Q&A Addresses Data Quantity and Quality
In the Q&A after her presentation, Kahn addressed questions regarding data quantity and quality.
Question: “You said earlier as knowledge increases, uncertainty decreases. How much weight do you put behind that during an inspection? It seems to me that companies monitor everything and document everything and have thousands of documents.”
Answer: “That relates to what I meant about the quality of the data. I would ask first, ‘what do you want this to do for you? And is it doing what you want it to do?’ I see firms that categorize and track and trend a million types of topics, but it still isn’t helping them deal with some of their specific issues. I feel like they are too much in the weeds in some ways.”
“The flip side is there are people who are not monitoring anything. I have seen it both ways. I hear what you are saying, and I do not know what the solution is. But I do offer that feedback to firms. I have seen the issue with almost every firm I have been in.”
“What I evaluate is whether the data the firm is collecting is effective for the quality unit. Is it doing what it needs to be doing? If I am on-site and I am trying to track and trend an issue that I see, and they can’t even provide me with the raw data to answer that question—if I can’t do it, they can’t do it. Which means they haven’t been doing it. To me, regardless of how much is there, it is all about the quality of the data.”
[NOTE: Are you looking for enforcement records that cite a specific keyword? Get a free 483 Observation Report that analyzes 25,000+ 483s for the FDA issue you’re looking for.]