Systematic Reviews
Role of the librarian in this stage
A librarian can advise you on quality assessment for your systematic review, including:
- What the quality assessment or risk of bias stage of the review entails
- How to choose an appropriate quality assessment tool
- Best practices for reporting quality assessment results in your review
Contact HSL About Your Review
Ready to start a systematic review? HSL Librarians can help!
Fill out the Systematic Review Request Form and the best-suited librarian will get back to you promptly.
Types of bias
After the screening process is complete, the systematic review team must assess each article for quality and bias. There are various types of bias, some of which are outlined in the table below from the Cochrane Handbook.
Cochrane Handbook - Table 10.1.a: Definitions of some types of reporting biases
| Type of reporting bias | Definition |
|---|---|
| Publication bias | The publication or non-publication of research findings, depending on the nature and direction of the results |
| Time lag bias | The rapid or delayed publication of research findings, depending on the nature and direction of results |
| Multiple (duplicate) publication bias | The multiple or delayed publication of research findings, depending on the nature and direction of results |
| Location bias | The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results |
| Citation bias | The citation or non-citation of research findings, depending on the nature and direction of the results |
| Language bias | The publication of research findings in a particular language, depending on the nature and direction of the results |
| Outcome reporting bias | The selective reporting of some outcomes but not others, depending on the nature and direction of the results |
Using Covidence for Quality Assessment
Covidence uses Cochrane Risk of Bias (which is designed for rating RCTs and cannot be used for other study types) as the default tool for quality assessment of included studies. You can opt to manually customize the quality assessment template and use a different tool better suited to your review. More information about quality assessment using Covidence, including how to customize the quality assessment template, can be found below. If you decide to customize the quality assessment template, you cannot switch back to using the Cochrane Risk of Bias template.
-
Covidence FAQs on Quality AssessmentCommonly asked questions about quality assessment using Covidence
-
Covidence YouTube ChannelA collection of Covidence-created videos
Systematic Reviews and Meta-Analyses
-
JBI Critical Appraisal Checklist for Systematic Reviews and Research SynthesesAn 11-item checklist for evaluating systematic reviews
-
AMSTAR ChecklistA 16-question measurement tool to assess systematic reviews
-
AHRQ Methods Guide for Effectiveness and Comparative Effectiveness ReviewsThe AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews, provides detailed information on selecting eligibility criteria, searching the literature, data extraction, quality assessment, and other steps in the creation of a systematic review.
-
CASP - Systematic ReviewA checklist for quality assessment of systematic review from the Critical Appraisal Skills Programme
-
Scottish Intercollegiate Guidelines Network (SIGN)Checklists for quality assessment
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
Clinical Practice Guidelines
-
National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) InstrumentA 15-item instrument using a scale of 1-5 to evaluate a guideline's adherence to the Institute of Medicine's standard for trust worth guidelines
-
AGREE-II Appraisal of Guidelines for Research and EvaluationThe Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument evaluates the process of practice guideline development and the quality of reporting
Randomized Controlled Trials (RCTs)
-
Cochrane Risk of Bias (ROB) 2.0 ToolTemplates are tailored to randomized parallel-group trials, cluster-randomized parallel-group trails (including stepped-wedge designs), and randomized cross-over trails and other matched designs.
-
CASP- Randomized Controlled Trial Appraisal ToolA checklist for RCTs created by the Critical Appraisal Skills Program (CASP)
-
The Jadad ScaleThe Jadad scale assesses the quality of published clinical trials based methods relevant to random assignment, double blinding, and the flow of patients.
-
CEBM-RCTA critical appraisal tool for RCTs from the Centre for Evidence Based Medicine (CEBM)
-
Checklist for Randomized Controlled Trials (JBI)A critical appraisal checklist from the Joanna Briggs Institute (JBI)
-
Scottish Intercollegiate Guidelines Network (SIGN)Checklists for quality assessment
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
Cohort Studies
-
CASP- Cohort StudiesA checklist created by the Critical Appraisal Skills Programme (CASP) to assess key criteria relevant to cohort studies
-
Checklist for Cohort Studies (JBI)A checklist for cohort studies from the Joanna Briggs Institute
-
The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analysesValidated tool for assessing case-control and cohort studies.
-
Scottish Intercollegiate Guidelines Network (SIGN)Checklists for quality assessment
-
STROBE ChecklistA checklist for quality assessment of case-control, cohort, and cross-sectional studies
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
Case-Control Studies
-
CASP- Case Control StudyA checklist created by the Critical Appraisal Skills Programme (CASP) to assess key criteria relevant to case-control studies
-
STROBE ChecklistA checklist for quality assessment of case-control, cohort, and cross-sectional studies
-
Tool to Assess Risk of Bias in Case Control Studies by the CLARITY Group at McMaster UniversityA quality assessment tool for case-control studies from the CLARITY Group at McMaster University
-
Scottish Intercollegiate Guidelines Network (SIGN)Checklists for quality assessment
-
The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analysesValidated tool for assessing case-control and cohort studies.
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
-
JBI Checklist for Case-Control StudiesA checklist created by the Joanna Briggs Institute
Cross-Sectional Studies
-
STROBE ChecklistA checklist for quality assessment of case-control, cohort, and cross-sectional studies
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
Diagnostic Studies
-
CASP- Diagnostic StudiesA checklist for diagnostic studies created by the Critical Appraisal Skills Program (CASP)
-
QUADAS-2A quality assessment tool developed by a team at the Bristol Medical School: Population Health Sciences at the University of Bristol
-
Critical Appraisal Checklist for Diagnostic Test Accuracy Studies (JBI)A checklist for quality assessment of diagnostic studies developed by the Joanna Briggs Institute
-
Scottish Intercollegiate Guidelines Network (SIGN)Checklists for quality assessment
Economic Studies
-
Consensus Health Economic Criteria (CHEC) List19 yes-or-no questions, one for each category to assess economic evaluations
-
CASP- Economic EvaluationA checklist for quality assessment of economic studies by the Critical Appraisal Skills Programme
-
Scottish Intercollegiate Guidelines Network (SIGN)Checklists for quality assessment
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
Mixed Methods
-
McGill Mixed Methods Appraisal Tool (MMAT) 2018 User GuideSee full site (public wiki link above) for additional information, including FAQ's, references and resources, earlier versions, and more.
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
Qualitative Studies
-
CASP- Qualitative Studies10 questions to help assess qualitative research from the Critical Appraisal Skills Programme
-
LEGEND Evidence Evaluation ToolsA series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.