In September, the FDA released a draft of its long-awaited guidance on Computer Software Assurance for Production and Quality System Software. While this guidance is not final, it gives tech and quality teams insight into the agency's thinking that could lead to lower compliance risk and development time, while continuing to protect patient safety and product quality. Read on to reap the benefits of CSA.
- The draft guidance in summary — I've digested 25 pages cross-referencing numerous auxiliary regulations into this three-page explainer.
- An analysis of the guidance — what has changed, and where is the FDA headed next?
- The guidance in context — what are the other regulations you need to know, and how do they relate?
- An informed critique — what the FDA is missing, and what we want to see in a final guidance.
Medical device innovation is creating new possibilities in healthcare: diagnosing disease, facilitating clinical decision-making, and even treating patients. While exciting, there is risk in the nascent technology: remember when cybersecurity risk forced Medtronic to recall its insulin pumps, or when software failure diminished Dräger's battery power supply rapidly enough to merit recall. As software becomes increasingly prominent in medical devices and their production, their quality systems need an update.
For the last 20 years, the primary FDA guidance on the topic has been General Principles of Software Validation, an extensive set of best practices to establish software quality and safety. Over its significant lifespan, the guidance has witnessed the evolution of increasingly agile development in contemporary medical device software, where rapid, small changes continuously improve quality — an environment it does not sufficiently address. The mismatch has led many industry development processes to ossify into slower-than-necessary waterfall cycles and documentation-intensive approaches — to the detriment of progress. The FDA wants to maintain patient safety while continuing to support innovation in health tech.
New FDA guidance: Computer Software Assurance
In September the FDA published a draft of what could become the latest guidance. If the FDA gets this right, updates have the potential to accelerate treatment development by eliminating unnecessary testing and documentation efforts, and to increase patient safety by focusing effort where risk is greatest.
In the draft, Computer Software Assurance for Production and Quality System Software, the FDA outlines a risk-based approach for establishing and maintaining confidence that software is fit for its intended use. This method, Computer Software Assurance (CSA), aims to help companies invest their software assurance efforts — such as testing, monitoring, and quality controls — in proportion to the risk posed by potential failure of that software. An upshot the FDA emphasizes: the burden of validation is no more than necessary to address the risk.
The FDA outlines three steps in CSA to help software developers make risk-based decisions:
- Calculate the risk posed to patients if the software fails,
- Execute appropriate assurance activities to check the quality of the software, based on the potential risks, and
- Document the assurance steps taken with each piece of software.
Let's dive into the details of each of these steps.
- Medical device risk
- The potential for a device to harm the patient or user (decrease safety).
- Process risk
- The potential to compromise production or the quality system.
- High process risk
- Such a risk that foreseeably compromises safety — for example, maintaining process parameters that affect product properties identified as essential to device safety, or determining acceptability of a product or process with limited or no additional human awareness or review.
Calculating risk: consider reasonably foreseeable failures
The draft recommends we consider all reasonably foreseeable failures — and the risks resulting from each — based on the intended use, level of human oversight, and importance of each piece of software.
Intended use
Software intended for use directly in production or the quality system is likely higher risk than supporting software. For example, the software controlling the 3D printer that constructs our device (direct use in production) is likely higher risk than the software that provides a convenience notification to personnel to restock the printer with material (supporting production).
Human oversight
Risk is generally lower when software use is overseen by a human who can mitigate the impact of its failure. For example, an ERP system that automates stocking of essential device materials for which a human conducts a quality review may be considered low-risk, while an ERP system that automates stocking and inspection of the same materials without human review may be considered high-risk.
Essential to device safety or quality
Of course, risks will generally be higher to the extent that they impact safety or quality. For example, the risk of a software medical device presenting a misleading diagnosis will likely be higher than it reporting that the diagnosis is uncertain, and certainly higher than the risk of it failing to display a help menu or a release-notes dialog.
FDA is particularly concerned with safety. Software for which failure could foreseeably cause harm to patients or users is considered high-risk, while all other software is likely not.
FDA encourages us to consider the granularity at which we evaluate software risk and assurance effort. For a simple system, we might save time by performing a single risk evaluation for the system as a whole. However, if we decompose our software into individual features, we may be able to tune assurance efforts more efficiently to the varied risks and development lifecycles of each feature.
For example, consider an ERP system intended to automate stocking and inspection of materials without human review. A system-level risk evaluation may indicate "high-risk," leading to significant assurance efforts for all of its features. A feature-level risk evaluation may instead indicate "low-risk" for automated material stocking and "high-risk" for automated material inspection, potentially leading to a lower overall assurance effort because less is necessary for the low-risk feature.
Appropriate assurance
The draft recommends a risk-based testing approach, meaning that in general higher-risk software should be tested more rigorously and its testing should be documented more thoroughly. For high-risk features, consider full functional (end-to-end) coverage with auditability and traceability to requirements. For a modern software team accustomed to CI/CD, this could mean slowing down development cycles to lock high-risk release candidates for manual functional testing, or else automating those functional tests (with a system that may itself be high-risk). Reach out if this is a challenge your team is facing — I specialize in these solutions.
It further recommends that we consider any additional quality system controls that may decrease the impact of software failure on safety or quality. These controls may justify reduced assurance effort, for example:
- Consider software that automates anomaly detection in a production process from a production data source. An established data-integrity review process may justify reducing testing rigor for the software's handling of certain invalid data inputs.
- A software team might justify relying on unit testing alone for a feature if a separate QA team will perform full functional testing.
It also notes that assurance activities for production and quality system software often inherently cover the performance of supporting software, which could reduce the need for additional assurance activities for that supporting software. For example, the assurance process for a spreadsheet function that produces performance statistics for a curing process based on temperature readings may involve entering example temperature readings — in the process of which we may also sufficiently test the supporting function of recording temperature readings.
Finally, remember that the FDA is most concerned with high-risk software (software for which failure could affect safety). Focus attention on identifying these risks, testing fully, and provisioning objective evidence.
Documentation
The draft recommends "manufacturers document their decision-making process for determining whether [software] is intended for use as part of production or the quality system in their Standard Operating Procedures (SOPs)." In this spirit, I recommend documenting your risk-assessment and level-of-assurance processes — e.g. how you get from a piece of software to a "high" or "low" risk, and from there to appropriate assurance efforts.
In documenting assurance activities themselves, it gives concrete suggestions:
- Intended use of the software
- Determination of risk
- Assurance activities conducted, including:
- Description of testing commensurate with robustness (robust includes: objectives, cases, and independent review and approval of test cases)
- Issues found
- A conclusion statement declaring acceptability of the results
- Date and name of tester
- Review and approval when appropriate (e.g. signature and date when necessary)
Engineering readers may note the readiness of such a record to be automated, as they should. The draft looks favorably on assurance automation, positing that manufacturers will "leverage automated traceability, testing, and the electronic capture of work performed to document the results, reducing the need for manual or paper-based documentation." Development teams following best practices like automated testing, version control, and managed infrastructure likely have many of the needed quality records already. Get credit for this quality work you're already doing by including these practices in your SOPs and storing the resulting quality records in a Part 11-compliant manner.
In keeping with the least-burdensome approach, the draft is explicit that "documentation of assurance activities need not include more evidence than necessary to show that the software … performs as intended for the risk identified." As a rule of thumb, ask whether your documentation gives you a baseline for improvements and a reference point if issues occur in the future.
Who should use CSA
This draft narrowly targets software used to produce medical devices or establish the quality of those devices. For example, a manufacturer might use:
- A spreadsheet to document temperature readings throughout a production process, and to produce performance statistics from those readings to monitor the process.
- An Enterprise Resource Planning system to automate material stocking, inspect those materials for quality characteristics, or handle delivery of the produced device to its user.
- A learning management system to manage, record, track, and report on employee training.
- A proprietary system to control a 3D printer, test its input material quantity and quality, perform functional testing on a sample of its products, and persist production parameters for use in quality monitoring and analysis over time.
- Off-the-shelf developer tools to assist its software team in producing a software medical device.
- A proprietary system to test and deploy a cloud-based software medical device.
Although this guidance would cover only production and quality-system software — and not software medical devices themselves — I suspect it is only the first in a series of updates that will ultimately transition all FDA software guidance to a CSA approach.
Of course, some software will likely forever remain out of scope for this guidance, including:
- Software intended for management of general business processes or operations, such as email or accounting applications.
- Software intended for establishing or supporting infrastructure not specific to production or the quality system, such as networking or continuity of operations.
The goal
The goal is to maintain patient safety while continuing to support innovation in health tech. We've long awaited an update that would focus validation effort where it's most needed, reduce the burden where it isn't, and help teams validate appropriately in more agile environments.
Where to learn more
- The referenced draft guidance: Computer Software Assurance for Production and Quality System Software.
- The FDA's 2002 validation guidance, section 6 of which would be superseded by the content in this draft: General Principles of Software Validation.
- 21 CFR Part 820: Quality System Regulation.
Wrangling software validation in a regulated build?
I help health-tech teams shift quality evidence left into the pipeline — so compliance becomes a byproduct of building well, not a tax on it. If risk-based assurance is a challenge you're facing, let's talk.
Get in touch