Clinical evaluation is a cornerstone of compliance under the EU Medical Device Regulation (MDR), directly tied to the safety and performance of a medical device throughout its lifecycle.
Yet, despite its importance, non-conformities in clinical evaluation reports remain one of the most common reasons for delays, rejections, and costly remediation work during conformity assessment.
When a clinical evaluation falls short - whether due to vague safety and performance objectives, inadequate literature appraisal, or poorly documented search methods - manufacturers face serious consequences. These may include prolonged Notified Body reviews, increased regulatory scrutiny, and even market withdrawal of devices.
Addressing clinical evaluation non-conformities isn’t just about avoiding negative outcomes; it’s about building a robust, evidence-based foundation that demonstrates your device meets essential safety and performance requirements. For manufacturers, mastering this process means faster approvals, fewer surprises during conformity assessments, and ultimately, safer outcomes for patients.
This video series breaks down the most common clinical evaluation non-conformities, explains why they keep happening, and most importantly, shows you how to fix (or avoid) them for good.
Episode 3: “It is not clear that systematic search methods have been used for the literature review”
Whether you’re new to clinical evaluation or refining an existing process, this video will help you design a robust and transparent search strategy that aligns with MDR expectations.
In this episode, we’ll break down:
- What a systematic search method is
- Key components of a systematic search
- How systematic search methodology should be applied during Clinical Evaluation
By the end of the episode, you’ll be able to confidently design and implement a systematic literature search for both State-of-the-art (SOTA) devices and the Device Under Evaluation (DUE) when conducting Clinical Evaluations.
At Mantra Systems, we specialise in delivering high-quality, MDR-compliant Clinical Evaluation Reports that stand up to Notified Body scrutiny. Whether you need a full CER, a literature review strategy, or expert input on addressing non-conformities, our team is here to help. Contact us today for a free, no obligation consultation on how we can support your clinical evaluation needs.
Missed previous episodes? Catch up here:
- Episode 1: “Safety and Performance objectives lack specific and measurable acceptance criteria”
- Episode 2: “Appraisal of literature sources has not been conducted properly”
Okay. Hi everyone and welcome to episode three in our ongoing series on how to fix clinical evaluation nonconformities. It’s Paul here from Mantra Systems. It’s my pleasure to take you through this latest installment of the series and uh let’s get to it.
So um as a reminder, this series is focused on addressing a key problem in um achieving success following submission of a clinical evaluation and that’s that nonconformities are actually very common under the MDR. But although it’s necessary to ensure that nonconformities are resolved, um they can be quite difficult to interpret and it may not always be clear how to fully address any nonconformities which arise, which in turn can lead to further rounds of review with increased costs, loss, time, stress, and worry.
And this series is intended to help you avoid common reasons for nonconformities. So, episode by episode, we’re working through common non-conformities one at a time and digging deep into what they mean and importantly, how to solve or avoid them in the first place. And where possible we tried to link to general principles for optimal conduct of clinical evaluation.
The Nonconformity in Focus
So this is episode three and this time we’re going to look at the following nonconformity:
It is not clear that systematic search methods have been used for the literature review.
So in order to unpack this one, we need to consider:
- What is a systematic search method?
- What are the components of a systematic search?
- How should systematic search methodology be applied during clinical evaluation?
Understanding those areas will help avoid this non-conformity arising.
What Is a Systematic Search Method?
Well, it is a structured and planned approach to finding relevant sources for literature reviews. And in this case we’re talking about its application to clinical evaluation.
From a clinical evaluation perspective, it’s relevant in two sectors:
- The literature search that’s conducted for the state-of-the-art.
- The search that needs to be done for the device under evaluation.
Ideally, they are two separate searches, documented and recorded separately.
Why It Matters
A systematic search method applied correctly helps ensure several things:
- Literature searches are conducted according to a validated prespecified plan. In other words, you’re not making it up as you go along.
- Literature searches have considered all relevant factors to ensure full coverage of clinical use of either state-of-the-art devices or the device under evaluation.
- Searches are reproducible so they can be conducted more than once and result in the same outcome.
- Bias and the risk of accidentally excluding relevant sources are minimized.
The Three Key Stages
On a very basic level, there are three key stages to a systematic literature search:
- Intended purpose
- Research questions
- Search terms
Intended Purpose
The intended purpose places the literature search into context and helps form an input to the definition of research questions.
Spending time on the intended purpose helps map out:
- The types of patients and conditions that fall under the scope of the search,
- The context of clinical use, and so on.
It is a vital input to the scope of the literature search and also shows traceability to how the device is used and applied in clinical practice. So it is an important albeit rather basic stage from the perspective of a literature search. But don’t skip it.
Research Questions
The next stage is to begin crafting research questions.
There’s lots of ways to do this, but ultimately a research question or a set of research questions just reduces what you want to achieve from the search to a list of simple questions. And it’s generally preferred that a validated format is used. PICO is one of them. We’re going to take a look at PICO in a moment.
It’s a good stage to explore and refine the scope of the search.
It’s very easy to change research questions—much easier than changing search terms.
Because with search terms, you have to rerun a search each time you change it, and that can be quite labor-intensive, especially if you’ve already started with inclusion/exclusion screening.
So research questions are worth spending time on because they form a vital input to the specific searches that you’ll run.
The PICO Framework
Let’s have a quick look at one validated method—it’s not the only one—but PICO is an important one.
PICO stands for:
- Patient
- Intervention
- Comparator
- Outcome
It’s just a way of breaking down the search or the scope of search into these specific stages.
Example:
- Patients: Patients with osteoarthritis of the hip who are candidates for total hip replacement surgery.
- Intervention: Ceramic-on-ceramic hip implants.
- Comparators: Metal-on-metal or metal-on-polyester hip implants.
- Outcomes: Pain VAS and range of motion.
That’s just one way of framing research questions using the PICO system.
Stage Three: Search Terms
What we’ve got here is a series of screen grabs just doing an example search on PubMed.
Example breakdown:
- Search 18: Osteoarthritis (produces many results)
- Search 19: Total hip replacement or hip arthroplasty
- Search 20: THR (abbreviation of total hip replacement)
You want to avoid repeating the same term over and over again, especially when using boolean AND, because it will look for multiple instances unnecessarily.
In PubMed’s advanced search:
- You can combine individual searches using boolean operators.
- Click the three dots next to a search and select ‘Add query’.
- Combine subsequent searches using AND.
Result:
- A refined composite search that significantly narrows down the results (e.g., to 51 highly relevant studies).
Additional Systematic Search Considerations
1. Database Specification
State in advance which databases will be used in your search protocol.
Examples:
- PubMed
- Google Scholar
- Embase
- Cochrane
- WDI
Usually, using a minimum of two is best practice. Otherwise, it may suggest bias.
2. Inclusion/Exclusion Screening
Refine your results to only those that are relevant.
- Define exclusion criteria in advance and document them in the search protocol.
- Prevents designing criteria reactively to include only favorable results.
Example:
- Code E1: Exclude in vitro, pre-clinical, or animal studies because their results may not transfer to humans.
This ensures every excluded result has a justification.
3. Handling Duplicate Results
Problems:
- Same study found more than once.
- Same dataset appearing in multiple publications.
Risks increase when including systematic reviews, which:
- May duplicate primary data sources.
- Can create overlap and confusion.
Tactics:
- Look out for recurring author names, shared recruitment techniques, and subsidiary publications.
- Consider excluding systematic reviews and focus on primary sources.
Including duplicates risks skewing your analysis by over-weighting repeated data.
4. Use of Filters
Search databases offer filters, but:
- Use with caution.
- Carefully justify their use.
- Filters vary across databases.
Common justifiable filter: time limits, especially for state-of-the-art searches.
For fast-evolving fields like software as a medical device, old studies may no longer be relevant.
5. Reproducibility
Another person should be able to:
- Rerun the search,
- Achieve the same results.
Application to Clinical Evaluation
- Create a detailed search protocol for both the SOT and device-specific reviews.
- Document:
- Research questions
- Search terms
- Exclusion criteria and justifications
- Justify use of filters.
- Remove duplicate results.
- Use validated methods.
- Ensure full reproducibility.
In summary, for Nonconformity Number Three, we’ve covered:
- What a systematic search method is,
- The components of a systematic search,
- How systematic search methodology should be applied during clinical evaluation.
As a reminder, Mantra Systems is a clinical evaluation specialist.
- Extensive experience across the market,
- Clinical evaluations for all classes of medical device,
- A current 100% submission success record.
Feel free to reach out to us if you have any questions about clinical evaluation or you’d like to speak with any of our specialists with regard to your own.
That concludes episode three of this non-conformity series.
Many thank yous for your kind attention.
