Systematic Reviews & Other Syntheses

Using a framework to develop your research question can help to identify the main concepts of your review topic. Whichever framework you use, try to identify the main concepts of your question and the synonyms / related terms that might be used to describe each of those concepts:

Image from UCSF Library: https://guides.ucsf.edu/c.php?g=126216&p=825824 

Combining search terms with AND or OR:

• Use AND to combine different key concepts of your search: childhood AND obesity. Narrows results
• Use OR to include similar terms / synonyms (and sometimes antonyms) for a concept: childhood OR adolescence. Fertility OR infertility. Broadens results

Considerations for searching:

It is not always necessary to include all of your main concepts in the search. For example, in some PICO questions, the outcome is implied and does not need to be included. In other cases, any and all outcomes might of interest and so the search strategy can leave this out to keep it open.

Research evidence:

• A research study that compared PICO and PICOS formats for systematic reviews of qualitative research found that PICO demonstrated higher sensitivity than PICOS (Methley et al., 2014). The researchers therefore recommend using PICO as the preferred format for systematic review searching and only using PICOS (which demonstrated higher specificity) when time and resources are limited.

Types of search terms

When conducting comprehensive database searches, the search terms for each concept should consist of both subject headings (indexed terms) where available, as well as keywords, in order to increase the sensitivity of the search.

• Subject headings are assigned to bibliographic records in a database by indexers in order to identify the main concepts of an article. Different databases use their own subject heading classification systems. See the page on Subject Heading Searching for more information.
   
• Keywords are words or phrases that can be searched for in different database fields such as title, abstract, author keywords, journal etc.
   

Strategies for identifying search terms

It can be extremely helpful if you already know of a few eligible studies for your synthesis. If the eligible studies have been indexed in databases such as MEDLINE and Embase you can review the subject headings that were applied to these studies to harvest a list of relevant search terms. Relevant studies can also be used to identify additional keywords that are relevant for the search.

Tools:

• Yale MeSH Analyzer allows searchers to enter relevant PMID numbers from MEDLINE/PubMed to analyze MeSH terms that might be relevant for building a comprehensive search strategy.
   

Considerations for selecting search terms

When conducting a synthesis, the search process is usually very comprehensive as most syntheses strive to find all studies that are relevant to the review topic.

• The search strategy should include as many relevant search terms as possible but within reason
• While it is not uncommon for synthesis searches to retrieve results in the thousands, reviewer fatigue during the screening process can increase the risk of missing relevant studies
• If you think the search strategy may be too broad, or that it is retrieving too many results, it may be appropriate to reconsider some of the search terms and/or search fields being used
• It is good practice to test different search strategies to see what articles would be missed if certain search terms are not included

Some syntheses will include the appropriate study design(s) as part of the search strategy (the 'S' in PICOS and PICoS above).

Identifying particular study designs in a database search is not as straight-forward as, for example, limiting to the publication type "Randomized Controlled Trials" in PubMed. This is partly because not all indexed records get assigned the appropriate publication type but also because databases like Ovid MEDLINE/PubMed include non-indexed citations that either have not or will not receive indexing to indicate the study design.

Search filters ("hedges") have been developed for different databases to do just that. These filters can be applied as published or they can be revised to increase the specificity or sensitivity of your search.
 

Study design filters:

Research evidence:

• DeJean, D., Giacomini, M., Simeonov, D., & Smith, A. (2016). Finding qualitative research evidence for health technology assessment. Qualitative Health Research, 26(10), 1307-1317.
   
• Waffenschmidt, S., Navarro-Ruan, T., Hobson, N., Hausner, E., Sauerland, S., & Haynes, R. B. (2020). Development and validation of study filters for identifying controlled non-randomized studies in PubMed and Ovid MEDLINE. Research synthesis methods, 11(5), 617–626. https://doi.org/10.1002/jrsm.1425

Publication format

It may also be tempting to exclude publication formats such as letters from your search to reduce the number of results.

Research evidence:

• A study by Iansavitchene et al. (2008) found that some RCTs are published as letters in MEDLINE (publication type Letter) and that systematic reviewers may miss potentially relevant studies by excluding letters in their search. Iansavitchene et al. explain that "Reviewers undertaking systematic reviews of RCTs should not exclude all letters, but rather use the logical construction NOT (letter.pt. NOT randomized controlled trial.pt.) when excluding letters from a MEDLINE search" (p. 715). This appears to be the case for editorials as well. Similarly, systematic reviewers should not exclude all editorials from a search.

If interested in adult populations, remove search results indexed as infant/child/adolescents only 

Note: For databases on the Ovid platform (MEDLINE, Embase, PsycINFO, etc.), the NOT operator can be manually entered by typing the last search line number into the search box and then copying and pasting the NOT statements below. In PubMed and CINAHL, the NOT operator can be applied in from the Search History.

If interested in child populations, remove search results indexed as adult only

Note: For databases on the Ovid platform (MEDLINE, Embase, PsycINFO, etc.), the NOT operator can be manually entered by typing the last search line number into the search box and then copying and pasting the NOT statements below. In PubMed and CINAHL, the NOT operator can be applied in from the Search History.

If your review topic implies that the research subjects are human, as is often the case with qualitative research, it is advisable to resist the temptation to limit to humans. However, if your search results retrieve a large number of animal studies, you may consider limiting to human studies.

Limiting to humans is not as straight-forward, for example, as using the limit for "Humans" in Ovid MEDLINE. This is partly because indexed records in a database may not have been assigned a subject heading for the appropriate species but also because databases like PubMed/Ovid MEDLINE include non-indexed citations. These records either have not or will not receive indexing to indicate whether the article discusses animals and/or humans.

The recommended method for limiting to human studies in a database is to exclude indexed records that have been assigned a subject heading for animals but that have not been assigned a subject heading for humans. This will remove indexed records that discuss animals only.

For example, this command would read as follows:

*Where the Search# represents the search line that you want to limit to human studies.

It is important to note that limiting to human studies is not equivalent to excluding animal studies since the latter would remove studies that discuss both humans and animals. Therefore do not use the command: Search# NOT animals/ (for example).

Language:

• Language limits should not be applied to search strategies that are intended to be as comprehensive as possible. Reviewers should attempt to identify all relevant studies, regardless of language, to reduce the likelihood of publication bias. If it is not possible to have non-English-language studies translated, the review should report the number of non-English-language studies that were eligible but not included in the review. For more information, refer to the section on Language bias in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins, 2011).

Publication date:

• Date restrictions may be appropriate for updating previously published systematic reviews. Otherwise, "Date restrictions should be applied only if it is known that relevant studies could only have been reported during a specific time period, for example if the intervention was only available after a certain time point" (Higgins, 2011, Section 6.4.9).

Research evidence:

• Sampson and McGowan (2006) examined 63 MEDLINE systematic review search strategies to investigate common errors made by reviewers. They found that 82.5% of the assessed search strategies contained errors that could potentially lower recall of relevant studies.
  

The most common search errors reported were:

• Missed subject headings/MeSH terms (44.4%)
  
• Unwarranted explosion of MeSH terms (38.1%)
  
• Irrelevant MeSH or free text terms (28.6%)
  
• Missed spelling variants (20.6%)
  
• Failure to tailor the search strategy for other databases (20.6%)
  
• Improper use of AND, OR and NOT to combine searches (19.0%)

Tip!

• When executing your search strategy in different databases, only search one term/phrase per search line and combine searches accordingly afterwards. This way, if you misspell a word or use an improper subject heading, the number of results retrieved for that search line should be an indication that an error has occurred. Note that not all misspelled words will retrieve 0 results as there are some instances where terms are misspelled in the citation information.

Grimshaw J. A guide to knowledge synthesis: a knowledge synthesis chapter. 2010:1-56. http://www.cihr-irsc.gc.ca/e/41382.html.

Iansavichene, A. E., Sampson, M., McGowan, J., & Ajiferuke, I. S. (2008). Should systematic reviewers search for randomized, controlled trials published as letters?. Annals of Internal Medicine, 148(9), 714-715.

Lefebvre C., Glanville J., Briscoe S., Littlewood A., Marshall C., Metzendorf M.-I., Noel-Storr A., Rader T., Shokraneh F., Thomas J. & Wieland L.S. Chapter 4: Searching for and selecting studies. In: Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J. & Welch V.A. (Eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019. Available from www.training.cochrane.org/handbook.

Stern, C., & Kleijnen, J. (2020). Language bias in systematic reviews: you only get out what you put in. JBI evidence synthesis, 18(9), 1818–1819. https://doi.org/10.11124/JBIES-20-00361