Skip to main content

Systematic Reviews & Other Syntheses

Introduction

This page provides some basic considerations for developing comprehensive search strategies. Both inexperienced and experienced searchers are encouraged to consult with a librarian when conducting a synthesis search.

Identifying Search Concepts

Quantitative and/or qualitative reviews:

Quantitative and/or qualitative research questions often utilize the PICO(S) format to identify the key concepts of the question.

PICO(S):

P Patient, Population or Problem
I Intervention
C Comparator/Control
O Outcomes
(S)    Study design


Considerations:

  • The PICO(S) concepts that you include in your search strategy are generally combined using 'AND' so that each of these concepts will appear in the results.

    For example:   P   AND   I   AND   S         -or-         P   AND   I   AND   C
     
  • Some reviews may actually use 'OR' to combine the intervention (I) and comparator (C) in order to increase the sensitivity of the search, particularly if the search is retrieving very few results. While this approach is more comprehensive it's not often practical because of the number of additional irrelevant results it can retrieve.

    For example:    P   AND   (I   OR   C)   AND   S
     
  • Combining more of the PICO(S) concepts (with 'AND') will narrow your search and reduce the number of results retrieved. However, every PICO(S) concept incorporated into the search strategy in addition to P and I will increase your risk of missing relevant studies.
     
  • The PICO format does not always lend itself well to qualitative research. As Cooke et al. (2012) explain:"...the “Comparison” (C) is not typically part of a qualitative research question so becomes irrelevant, whereas both “Intervention” (I) and “Outcome” (O) might need to be manipulated to fit with the qualitative paradigm" (p. 1437). For this reason, reviewers may prefer to use PICo for qualitative research (see below).

Guidelines:

  • If your topic retrieves a low number of search results just by combining P and I, you may decide not to include any other PICO(S) concepts in the search strategy in order to increase the sensitivity of the search.
  • It is not necessary to include the comparator/control (C) concept in the search strategy if you are interested in comparing the intervention with all other alternatives (standard practice, control, no intervention, other interventions etc.).
     
  • Studies of healthcare interventions will assess some type of outcome. If you are interested in any or all measured outcomes then the search strategy does not need to include the outcome concept (O) since it is already implied. If you are only interested in specific outcomes to which there are other alternatives, you may decide to include the outcome concept in the search strategy, particularly if you are trying to reduce the number of results.

Research evidence:

  • A research study that compared PICO and PICOS formats for systematic reviews of qualitative research found that PICO demonstrated higher sensitivity than PICOS (Methley et al., 2014). The researchers therefore recommend using PICO as the preferred format for systematic review searching and only using PICOS (which demonstrated higher specificity) when time and resources are limited.

Qualitative reviews:

For qualitative research, PICO (presented above) or PICo formats may be utilized to identify the key concepts of your question.

PICo:

P     Population / types of Participants
I phenomenon of Interest
Co Context


Considerations:

  • There is no specific requirement for an outcome to be included in qualitative results so this concept may be left out.
  • The PICo concepts would be combined using 'AND'.

    For example:   P   AND    I   AND   Co
     
  • Types of studies may be considered in addition to the PICo concepts > PICo(S).
     
  • For a complete overview of PICo, refer to pages 22-24 of the Joanna Briggs Institute Reviewers' Manual, 2011 edition.

Research evidence:

  • The SPIDER format is not recommended for systematic reviews of qualitative research as two separate studies have reported that this method is not as sensitive as PICO resulting in relevant studies being missed (Cooke et al., 2012; Methley et al., 2014).

Identifying Search Terms

Types of search terms:

  • When conducting comprehensive database searches, the search terms for each concept should consist of both subject headings (indexed terms) where available, as well as keywords, in order to increase the sensitivity of the search.
     
    • Subject headings are assigned to bibliographic records in a database by indexers in order to identify the main concepts of an article. Different databases use their own subject heading classification systems.
       
    • Keywords are words or phrases that can be searched for in different database fields such as title, abstract, author keywords, journal etc.


Strategies for identifying search terms:

  • It can be extremely helpful if you already know of a few eligible studies for your synthesis. If the eligible studies have been indexed in databases such as MEDLINE and Embase you can review the subject headings that were applied to these studies to harvest a list of relevant search terms. Relevant studies can also be used to identify additional keywords that are relevant for the search.

Tools

  • Yale MeSH Analyzer allows searchers to enter relevant PMID numbers from MEDLINE/PubMed to analyze MeSH terms that might be relevant for building a comprehensive search strategy.
     
  • MedTerm Search Assist is a tool for identifying additional relevant terms (MeSH and keywords).


Considerations for selecting search terms:

  • When conducting a synthesis, the search process is usually very comprehensive as most syntheses strive to find all studies that are relevant to the review topic. The search strategy should include as many relevant search terms as possible but within reason. While it is not uncommon for syntheses searches to retrieve results in the thousands, reviewer fatigue during the screening process can increase the risk of missing relevant studies. If you think the search strategy may be too broad, or that it is retrieving too many results, it may be appropriate to reconsider some of the search terms and/or search fields being used. It is good practice to test different search strategies to see what articles would be missed if certain search terms are not included.

Applying Filters for Study Designs

Some syntheses will include the appropriate study design(s) as part of the search strategy (the 'S' in PICOS and PICoS above).

Identifying particular study designs in a database search is not as straight-forward as, for example, limiting to the publication type "Randomized Controlled Trials" in PubMed. This is partly because not all indexed records get assigned the appropriate publication type but also because databases like Ovid MEDLINE/PubMed include non-indexed citations that either have not or will not receive indexing to indicate the study design.

Search filters ("hedges") have been developed and tested for different databases to do just that. These filters can be applied as published or they can be revised to increase the specificity or sensitivity of your search:

Publication format:

It may also be tempting to exclude certain publication formats from your search to reduce the number of results (e.g. letters, editorials, reviews etc.).

Research evidence:

  • A study by Iansavitchene et al. (2008) found that some RCTs are published as letters in MEDLINE (publication type Letter) and that systematic reviewers may miss potentially relevant studies by excluding letters in their search. Iansavitchene et al. explain that "Reviewers undertaking systematic reviews of RCTs should not exclude all letters, but rather use the logical construction NOT (letter.pt. NOT randomized controlled trial.pt.) when excluding letters from a MEDLINE search" (p. 715). This appears to be the case for editorials as well. Similarly, systematic reviewers should not exclude all editorials from a search.

Limiting to Human Studies

If your review topic implies that the research subjects are human, as is often the case with qualitative research, it is advisable to resist the temptation to limit to humans. However, if your search results retrieve a large number of animal studies, you may consider limiting to human studies.

Limiting to humans is not as straight-forward, for example, as using the limit for "Humans" in Ovid MEDLINE. This is partly because indexed records in a database may not have been assigned a subject heading for the appropriate species but also because databases like PubMed/Ovid MEDLINE include non-indexed citations. These records either have not or will not receive indexing to indicate whether the article discusses animals and/or humans.

The recommended method for limiting to human studies in a database is to exclude indexed records that have been assigned a subject heading for animals but that have not been assigned a subject heading for humans. This will remove indexed records that discuss animals only.

For example, this command would read as follows:

Ovid MEDLINE *Search# NOT (exp animals/ NOT humans/)
Ovid Embase *Search# NOT ((exp animal/ or nonhuman/) NOT exp human/)


*Where the Search# represents the search line that you want to limit to human studies.

It is important to note that limiting to human studies is not equivalent to excluding animal studies since the latter would remove studies that discuss both humans and animals. Therefore do not use the command: Search# NOT animals/ (for example).

Limiting by Language and Date

Language:

  • Language limits should not be applied to search strategies that are intended to be as comprehensive as possible. Reviewers should attempt to identify all relevant studies, regardless of language, to reduce the likelihood of publication bias. If it is not possible to have non-English-language studies translated, the review should report the number of non-English-language studies that were eligible but not included in the review. For more information, refer to the section on Language bias in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins, 2011).

Publication date:

  • Date restrictions may be appropriate for updating previously published systematic reviews. Otherwise, "Date restrictions should be applied only if it is known that relevant studies could only have been reported during a specific time period, for example if the intervention was only available after a certain time point" (Higgins, 2011, Section 6.4.9).

Avoiding Common Search Errors

Research evidence:

  • Sampson and McGowan (2006) examined 63 MEDLINE systematic review search strategies to investigate common errors made by reviewers. They found that 82.5% of the assessed search strategies contained errors that could potentially lower recall of relevant studies.

The most common search errors reported were:

  • Missed subject headings/MeSH terms (44.4%)
  • Unwarranted explosion of MeSH terms (38.1%)
  • Irrelevant MeSH or free text terms (28.6%)
  • Missed spelling variants (20.6%)
  • Failure to tailor the search strategy for other databases (20.6%)
  • Improper use of AND, OR and NOT to combine searches (19.0%)

Tip!

  • When executing your search strategy in different databases, only search one term/phrase per search line and combine searches accordingly afterwards. This way, if you misspell a word or use an improper subject heading, the number of results retrieved for that search line should be an indication that an error has occurred. Note that not all misspelled words will retrieve 0 results as there are some instances where terms are misspelled in the citation information.

Bibliography