| Searching for studies may retrieve hundreds or thousands of references that will then need to be screened to identify eligible studies. Systematic review software programs like Covidence can help facilitate the process of removing duplicate references and the two-stage screening process. |
Searching multiple databases for studies will inevitably result in retrieving duplicate references. Removing duplicate references before screening will expedite the process.
There are a growing number of review software programs that can assist with de-duplicating references, including proprietary options such as Covidence and DistillerSR, and freely available options like Rayyan.
Queen's University has an institutional subscription to Covidence systematic review software. When you import search results from different databases into Covidence, it will automatically identify and remove duplicate references.
Traditionally, many researchers have attempted to use citation managers to de-duplicate references. However, citation management software was not designed with systematic reviews in mind, and user oversight is often built into the process. This can mean reviewing hundreds or thousands of references to confirm duplicates. It is therefore recommended that Queen's researchers utilize Covidence for removing duplicate references.
Why does more than one reviewer need to screen?
Having more than one reviewer independently assess citations for inclusion is one method of reducing the risk of biased decisions on study inclusion (Agency for Healthcare Research and Quality).
As the Institute of Medicine explains “[e]ven when the selection criteria are prespecified and explicit, decisions on including particular studies can be subjective (p. 110).”
Research evidence:
A recent study found that using a complete dual review approach, where two reviewers screen at both stages, identified additional eligible studies at both the title/abstract and the full-text stage (Stoll et al., 2019).
Guidelines:
Cochrane Handbook (2019):
4.6.4 Implementation of the selection process: Decisions about which studies to include in a review are among the most influential decisions that are made in the review process and they involve judgement. Use (at least) two people working independently to determine whether each study meets the eligibility criteria.
Agency for Healthcare Research and Quality (McDonagh et al., 2013):
Study selection process: Even with clear, precise inclusion criteria, elements of subjectivity and potential for human error in study selection still exist. For example, inclusion judgments may be influenced by personal knowledge and understanding of the clinical area or study design (or lack thereof).
The study selection process is typically done in two stages; the first stage involves a preliminary assessment of only the titles and abstracts of the search results. The purpose of this step is to eliminate efficiently all obviously ineligible publications. The second stage involves a careful review of the full-text publications.
Dual review—having two reviewers independently assess citations for inclusion—is one method of reducing the risk of biased decisions on study inclusion, as is recommended in the Institute of Medicine's “What works in healthcare: standards for systematic reviews.” Some form of dual review should be done at each stage to reduce the potential for random errors and bias. Reviewers compare decisions and resolve differences through discussion, consulting a third party when consensus cannot be reached. The third party should be an experienced senior reviewer. The two stages of assessment are discussed in more detail below. Dual review can help identify misunderstandings of the criteria and resolve them such that the studies included will truly fulfill the intended criteria.
Cicchetti, D. V., & Conn, H. O. (1976). A statistical analysis of reviewer agreement and bias in evaluating medical abstracts. Yale Journal of Biology and Medicine, 49(4), 373–383.
Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane, 2019.
Institute of Medicine. (2011). Finding what works in health care: standards for systematic reviews (S. Morton, L. Levit, A. Berg, & J. Eden, eds.). Washington, DC: The National Academies Press.
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017). Searching for studies: a guide to information retrieval for Campbell systematic reviews (p. 76).
McDonagh, M., Peterson, K., Raina, P., Chang, S., & Shekelle, P. (2013). Avoiding bias in selecting studies. In methods guide for effectiveness and comparative effectiveness reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US).
Stoll, C. R. T., Izadi, S., Fowler, S., Green, P., Suls, J., & Colditz, G. A. (2019). The value of a second reviewer for study selection in systematic reviews. Research Synthesis Methods, 10(4), 539–545. https://doi.org/10.1002/jrsm.1369
Westgate, M. J. (2019). revtools: An R package to support article screening for evidence synthesis. Research Synthesis Methods, 10(4), 606–614. https://doi.org/10.1002/jrsm.1374