Skip to Main Content

Medicine

Introduction

"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context. It is an essential skill for evidence-based medicine because it allows clinicians to find and use research evidence reliably and efficiently" (Burls, 2009).

Appraising Grey Literature

Grey literature is the term used to describe literature that is not formally published as a book or journal article (Higgins, 2011), including:

  • conference proceedings
  • preprints
  • dissertations and theses
  • governmental or private sector research
  • ongoing or unpublished clinical trial data
  • statistical publications
  • internal reports or working papers
  • technical reports

The critical appraisal checklists listed further up the page can be used to evaluate research study designs such as systematic reviews, RCTs, cohort studies, etc., regardless if the research is formally published or otherwise provided as a conference paper, pre-print, etc. Evaluating studies in the form of grey literature becomes more difficult when limited information is provided, which can be the case for conference abstracts, clinical trial data etc.

Grey Literature Checklists

  • The AACODS Checklist (Authority, Accuracy, Coverage, Objectivity, Date, Significance) was developed to appraise the widest range of grey literature sources possible (Tyndall, 2010)
     
  • The Ontario Agency for Health Protection and Promotion also offers the following considerations about the presence (reporting) and the appropriateness (validity) of the items described below may help to assess the quality of the source (2015):

 

 Reporting Validity
Author/ publisher Who is the author, are they who they claim to be?

Does the author/host have an agenda?

Publishing body:

  • Are they a well-known provincial/national government-affiliated public health agency?
  • For-profit company?
  • Political or philosophical agenda? (e.g. Fraser Institute vs. Canadian Centre for Policy Alternatives) 
Host/sponsor Does the hosting site have a sponsor?

Is the information shared for commercial purposes?

Does the site sponsor have a political or philosophical agenda? 
Facts and references Are references included? 

Are references to research evidence?

Is the information based on research/data/analysis or is it opinion piece?

Can you verify some cited sources, facts?

How current are the references? 
Currency Is a publication or revised date provided?  Is the information current?

Bibliography

Burls, A. (2009). What is critical appraisal? Hayward Medical Communications.

Ontario Agency for Health Protection and Promotion (2015). Public Health Ontario guide to appraising grey literature. Available from: https://www.publichealthontario.ca/-/media/documents/appraising-grey-lit-guide.pdf?la=en

Tyndall, J. (2010). AACODS Checklist. Flinders University. Available from http://dspace.flinders.edu.au/dspace/