Follow the Trail to Find Data Integrity Problems

How Do I Follow The Trail?

One of the most effective audits I ever participated in started with a systems-naive auditor asking a simple question: “Imagine I’m a sample arriving at your loading dock. What happens to me?”

We proceeded down the trail together, following the sample through the processes of accessioning, analysis, reporting, and storage.

While you can start at any point in the process, if you’re new to this, I recommend starting at the beginning. Figure out what should be happening and then find out what actually happens.

The shoulds…

It is often most effective to start with what should be happening, mapping out the process, and what’s happening to the data. Remember that the most important “should” is the predicate rule. Be sure you understand the predicate rule requirements for the process and the data.

What should the process be?

SOPs describe management’s intent for the process. Start by reviewing SOPs and sketching a trail map (in technical language, make a process map). You don’t need to be proficient with flow charting – you can do this with paper and pencil.

The main things you want to identify are:

• What are the inputs?
• Who does what?
• What are the outputs?

What should happen to the data?

As you map the SOPs, note:

• When data are created, modified, or deleted
• Who is authorized to perform these actions
• Watch the trail for warning signs
• What are some of the warning signs you might see?

Here’s an example: During an audit of a data management department, you find instructions in an SOP for performing hard deletes of data entered into an EDC system by clinical investigator site staff. A hard delete often means that nothing about the data is left behind in the database.

What do the predicate rules say? 21 CFR Part 312.62(b) requires the investigator to “prepare and maintain adequate and accurate case histories that record all observations.” ICH E-6 Section 5.5.3(c) requires there be “no deletion of entered data.” So when you see instructions for performing hard deletes, that’s a warning sign.

Prepare some questions:

• Who can perform hard deletes?
• When would they perform a hard delete?
• Is there an audit trail of the delete, including former values, time, date, and user?
• How would the investigator know data had been deleted?
• How are hard deletes documented in the system requirements, specifications, and testing?

Time for a dose of reality!

The Reality

It’s not enough to examine what should be happening in an audit. You have to do a reality check, and that means interviews and observations. Even if you’re doing a qualification audit, the provider should be able to give you a demonstration of their processes and systems.

What is the process?

Take a tour (not the warm, fuzzy 15 minute facility tour). This is your opportunity to engage with people responsible for executing the process and ask them to demonstrate how things really work. Have your map in hand and share it with the auditee. Ask for clarifications and take notes.

What is happening to the data?

During the tour, request user demonstrations of creating, modifying, and deleting data. Record the details: who did it and when? Ask for screenshots and reports from the system so you can verify what you saw matches what the system recorded.

After the tour, cross check what you saw with the process requirements and the system requirements, specifications, and testing.

As your confidence grows, you can go deeper and deeper into areas like user roles, data review, and configuration settings.

Final tip

It is not uncommon for reality to differ from the “shoulds.” Sometimes users will have developed undocumented workarounds for system or process limitations.

Believe it or not, they’re often just waiting for someone like you to come along and give them some traction to get the limitation fixed.

What are the Benefits?

Spending all your audit time up to your necks in validation binders is a bad thing.

The good things about following the trail are:

1. You will identify problems

I didn’t say risks, did I? A risk is the possibility that something bad will happen. A problem is a risk that has materialized. Your management needs and wants to know about risks and the detective and preventive controls that are in place and that are missing.

However, it’s even more important for them to know what the problems are so they can assure the appropriate corrective and preventive action plans are put in place.

2. Your audit team will be more effective

Your company probably uses audit teams for provider and internal process audits. Audit teams often include auditors who are highly skilled in GLP and GCP predicate rule requirements and auditors who have systems compliance expertise.

You can each sit at your own little corner of the table and do your own things. Or, you can follow the trail together. Working as a team, your individual strengths will combine to make for a highly effective audit.

Audits should provide information for decisions. If you follow the trail, you will reach this destination.

About the Author

Jamie Colgin is Govzilla’s GCP Product Manager and is the recipient of the prestigious Charles H. Butler Excellence in Teaching Award.  She joins Govzilla from Colgin Consulting, Inc.