FDA’s Quality Metrics Initiative, announced in April 2013 to help promote “quality-driven corporate culture” in drug and biologics manufacturing, has been slow in coming to fruition. However, its Quality Metrics Site Visit and Feedback Programs that began in earnest in 2019 may help breathe life back into that effort.
Find out more about what participants in those programs have shared regarding what the agency is looking for, what feedback has been given to FDA, and what progress has been made in a three-part series. Part 1 is below.
[Related: For the latest in pharma quality, download a FREE report featuring four articles by Jerry Chapman.]
At the International GMP Conference at the University of Georgia in Athens, Georgia, in early March 2020, co-sponsored by FDA, Procter & Gamble Quality Leader and Leadership Coach Steve Greer shared the results of FDA site visits to nine U.S. and European companies together with feedback gathered as part of his work with the International Society of Pharmaceutical Engineers (ISPE).
The FDA Site Visit Program provides an opportunity for in-depth engagement at the facility level as FDA observes the implementation of onsite quality metrics and quality culture programs. This enables FDA personnel to see how the data is gathered and used to drive decision-making as well as provides an opportunity to talk to employees from different functional areas of the company.
Figure 1 shows the types of sites FDA visited and the timing of those visits as presented by Greer. “You can see the range of types of sites that FDA visited, including high volume complex OTC sites,” Greer said. “You can also see that the experience I am going to relate comes from both the European and the U.S. perspective.”
Figure 1: Site Visit Types and Timeframes
“We were one of the crazy few who invited FDA to come in and visit. We said we would love to open up the kimono to show everybody everything,” Greer quipped. “And so, we opened up one of our sites. We have a site in the Carolinas. And in May of last year, we had six FDA folks show up.”
A couple of the visitors were from the compliance group, “which always makes leadership a little nervous. When it is just from the Office of Pharmaceutical Quality (OPQ) they are not too worried. But when compliance shows up, that always gets people interested,” Greer said.
‘We were one of the crazy few who invited FDA to come in and visit. We said we would love to open up the kimono to show everybody everything,’ Greer quipped.
“We had a really good visit and we shared a number of things. For those of you who were not able to participate, I want to give you a little summary of how those visits went. And I want to share some of the questions FDA asked us.” (Figure 2)
Figure 2 Sample of FDA Questions
Inventory Tops List of FDA Questions
“It might be interesting to know that one of the questions that came up was inventory,” Greer said. “For example, how do you manage your inventory? How do you measure your performance around inventory? And some of that is based on the fact that FDA is working very closely with the University of St. Gallen out of Switzerland, and part of their work has determined that on time in full—or OTIF—can be a very strong predictor of quality effectiveness.”
He noted that the OTIF measure may not be intuitive but pointed out that if the quality systems are working well, a company tends to provide its customers what they want when they want it. And in quality systems that are struggling, the firm often has trouble meeting those needs.
Other questions regarding how data is shared between sponsors and CMOs showed FDA’s interest in those relationships. Still, others investigated how metrics are defined and how quality culture is measured. “And this is one of the pieces that [FDA Office of Regulatory Affairs Office of Pharmaceutical Quality Operations (OPQO) Director Alonza Cruse] was talking about earlier in the conference regarding the NIPP program. The NIPP program asks questions around how effective the quality culture is at your site.”
They also asked which metrics were predictive in nature, looking forward rather than looking backward, and what the company considers the most important metrics.
“I am curious. How would you answer that last question? What would you say one of your most important metrics is?” Greer asked.
Audience member responses are in bold, and Greer’s brief remarks follow:
- Batches manufactured vs. batches released
- Right first time (i.e., doing what you said you would do)
- Out-of-specification findings overturned or invalidated (That is certainly one of the three that FDA has been focused on)
- Complaints. (What do those outside of our organization say about us and the quality of the product that we are making?)
Feedback Given to FDA
Greer shared some of the feedback the companies provided to FDA during the site visits. Industry indicated that it would like more of the following (detail on each follows):
- Co-creating the program with FDA rather than use the guidance/comment process
- Flexibility in definitions
- Context around the metrics
- Site-based metrics
- Assurance that metrics information will not be released through the Freedom of Information Act (FOIA)
Co-Create a Program
Industry indicated that it would like to share the journey more closely with FDA—that it would like to help create a program rather than use the guidance process to create it. The process today is that FDA creates guidance, gives it to industry and receives feedback from industry to the agency. The agency then creates revised guidance that goes back to industry, and then industry sends feedback on the revision to the agency.
“And so, we have been playing this kind of back and forth game. That is fine. It is good guidance practices,” Greer said. “But here, we would rather sit down at the table and co-create a program where we can pull academia, the agency, and industry together. So that is one of the points that we gave the agency.”
Flexibility in Definitions
Industry also requested that the agency allow some flexibility in definitions of terms. “How many people measure repeat deviations or something of that nature within your organization?” Greer asked.
“If you measure repeat or recurrent deviations, how many people struggle getting one plant to measure it the same way as another plant? So, even within our own companies, definitions can be a challenge, especially some that may be particularly important, like CAPA effectiveness and recurrent events.”
He also pointed to the challenges in standardizing definitions across the industry in the context of differences in supply chains, and the difficulty when attempting to take one definition and apply it across multiple product types.
‘And so, we have been playing this kind of back and forth game. That is fine. It is good guidance practices,’ Greer said. ‘But here, we would rather sit down at the table and co-create a program where we can pull academia, the agency, and industry together.’
Context
Similarly, the industry told FDA that context is particularly important. For example, “You cannot just take three measures and judge a site as good or bad based on those three measures,” Greer maintained.
Site-Based Metrics
The group also told FDA that they would prefer the metrics to be site-based rather than product-based. While product quality complaints need to be product-based, outside that, company metrics tend to be site-based today. “We judge a site, the culture of a site, and the performance of a site typically on a site scorecard or dashboard,” Greer said. “That is what a typical industry practice is. So, we suggested to FDA that site-based metrics are the preferred way of reporting.”
Not FOIA Releasable
The group also asked that the metrics remain confidential between FDA and each company, i.e., that it not be made available through FOIA.
“Those are the key messages that we shared with FDA,” Greer said. “There were obviously other messages, but I tried to summarize those that I thought were most important.”
In Part II of this series, Steve Greer discusses the FDA Quality Metrics Feedback Program from Procter & Gamble’s viewpoint as a participant in the program, including:
- How the program worked
- The kind of data his company submitted
- The incentives that were discussed to prompt firms to submit data to the program
- FDA’s Quality Maturity Program