Systematic Reviews: Step 6: Assess Quality of Included Studies
Created by Health Science Librarians
About Step 6: Assess Quality of Included Studies
In step 6 you will evaluate the articles you included in your review for quality and bias. To do so, you will:
- Use quality assessment tools to grade each article.
- Create a summary of the quality of literature included in your review.
This page has links to quality assessment tools you can use to evaluate different study types. Librarians can help you find widely used tools to evaluate the articles in your review.
If you reach the quality assessment step and choose to exclude articles for any reason, update the number of included and excluded studies in your PRISMA flow diagram.
Covidence includes the Cochrane Risk of Bias 2.0 quality assessment template, but you can also create your own custom quality assessment template.
A librarian can advise you on quality assessment for your systematic review, including:
Assess studies for quality and bias
After the screening process is complete, the systematic review team must assess each article for quality and bias. There are various types of bias, some of which are outlined in the table below from the Cochrane Handbook.
The most important thing to remember when choosing a quality assessment tool is to pick one that was created and validated to assess the study design(s) of your included articles.
For example, if one item in the inclusion criteria of your systematic review is to only include randomized controlled trials (RCTs), then you need to pick a quality assessment tool specifically designed for RCTs (for example, the Cochrane Risk of Bias tool).
Covidence Academy has a post on Common tools for risk of bias (quality) assessment.
Critically appraise included studies
Once you have gathered your included studies, you will need to appraise the evidence for its relevance, reliability, validity, and applicability.
Ask questions like:
Relevance:
- Is the research method/study design appropriate for answering the research question?
- Are specific inclusion / exclusion criteria used?
Reliability:
- Is the effect size practically relevant? How precise is the estimate of the effect? Were confidence intervals given?
Validity:
- Were there enough subjects in the study to establish that the findings did not occur by chance?
- Were subjects randomly allocated? Were the groups comparable? If not, could this have introduced bias?
- Are the measurements/ tools validated by other studies?
- Could there be confounding factors?
Applicability:
- Can the results be applied to my organization and my patient?
Select a quality assessment tool
What are Quality Assessment tools?
Quality Assessment tools are questionnaires created to help you assess the quality of a variety of study designs. Depending on the types of studies you are analyzing, the questionnaire will be tailored to ask specific questions about the methodology of the study. There are appraisal tools for most kinds of study designs. You should choose a Quality Assessment tool that matches the types of studies you expect to see in your results. If you have multiple types of study designs, you may wish to use several tools from one organization, such as the CASP or LEGEND tools, as they have a range of assessment tools for many study designs.
Click on a study design below to see some examples of quality assessment tools for that type of study.
- Cochrane Risk of Bias (ROB) 2.0 Tool
Templates are tailored to randomized parallel-group trials, cluster-randomized parallel-group trails (including stepped-wedge designs), and randomized cross-over trails and other matched designs. - CASP- Randomized Controlled Trial Appraisal Tool
A checklist for RCTs created by the Critical Appraisal Skills Program (CASP) - The Jadad Scale
A scale that assesses the quality of published clinical trials based methods relevant to random assignment, double blinding, and the flow of patients - CEBM-RCT
A critical appraisal tool for RCTs from the Centre for Evidence Based Medicine (CEBM) - Checklist for Randomized Controlled Trials (JBI)
A critical appraisal checklist from the Joanna Briggs Institute (JBI) - Scottish Intercollegiate Guidelines Network (SIGN)
Checklists for quality assessment - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- CASP- Cohort Studies
A checklist created by the Critical Appraisal Skills Programme (CASP) to assess key criteria relevant to cohort studies - Checklist for Cohort Studies (JBI)
A checklist for cohort studies from the Joanna Briggs Institute - The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses
A validated tool for assessing case-control and cohort studies - Scottish Intercollegiate Guidelines Network (SIGN)
Checklists for quality assessment - STROBE Checklist
A checklist for quality assessment of case-control, cohort, and cross-sectional studies - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- CASP- Case Control Study
A checklist created by the Critical Appraisal Skills Programme (CASP) to assess key criteria relevant to case-control studies - STROBE Checklist
A checklist for quality assessment of case-control, cohort, and cross-sectional studies - Tool to Assess Risk of Bias in Case Control Studies by the CLARITY Group at McMaster University
A quality assessment tool for case-control studies from the CLARITY Group at McMaster University - Scottish Intercollegiate Guidelines Network (SIGN)
Checklists for quality assessment - The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses
A validated tool for assessing case-control and cohort studies - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs. - Checklist for Case-Control Studies
A checklist created by the Joanna Briggs Institute
- STROBE Checklist
A checklist for quality assessment of case-control, cohort, and cross-sectional studies - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- CASP- Diagnostic Studies
A checklist for diagnostic studies created by the Critical Appraisal Skills Program (CASP) - QUADAS-2
A quality assessment tool developed by a team at the Bristol Medical School: Population Health Sciences at the University of Bristol - Critical Appraisal Checklist for Diagnostic Test Accuracy Studies (JBI)
A checklist for quality assessment of diagnostic studies developed by the Joanna Briggs Institute - Scottish Intercollegiate Guidelines Network (SIGN)
Checklists for quality assessment
- Consensus Health Economic Criteria (CHEC) List
19 yes-or-no questions, one for each category to assess economic evaluations - CASP- Economic Evaluation
A checklist for quality assessment of economic studies by the Critical Appraisal Skills Programme - Scottish Intercollegiate Guidelines Network (SIGN)
Checklists for quality assessment - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- McGill Mixed Methods Appraisal Tool (MMAT) 2018 User Guide
See full site for additional information, including FAQ's, references and resources, earlier versions, and more - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- CASP- Qualitative Studies
10 questions to help assess qualitative research from the Critical Appraisal Skills Programme - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- Critical Appraisal Checklist for Systematic Reviews and Research Syntheses
An 11-item checklist for evaluating systematic reviews - AMSTAR Checklist
A 16-question measurement tool to assess systematic reviews - AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews
A guide to selecting eligibility criteria, searching the literature, extracting data, assessing quality, and completing other steps in the creation of a systematic review - CASP - Systematic Review
A checklist for quality assessment of systematic review from the Critical Appraisal Skills Programme - Scottish Intercollegiate Guidelines Network (SIGN)
Checklists for quality assessment - LEGEND Evidence Evaluation Tools
A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.
- National Guideline Clearinghouse Extent of Adherence to Trustworthy Standards (NEATS) Instrument
A 15-item instrument using a scale of 1-5 to evaluate a guideline's adherence to the Institute of Medicine's standard for trust worth guidelines - AGREE-II Appraisal of Guidelines for Research and Evaluation
The Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument evaluates the process of practice guideline development and the quality of reporting
- NTACT Quality Checklists
Quality indicator checklists for correlational studies, group experimental studies, single case research studies, and qualitative studies developed by the National Technical Assistance Center on Transition (NTACT). (Users must make an account.)
A closer look at popular tools
Below, you will find a sample of four popular quality assessment tools and some basic information about each. For more quality assessment tools, please view the blue tabs in the boxes above, organized by study design.
| Tool | Study Design | About |
|---|---|---|
| Cochrane Risk of Bias 2.0 Tool | Randomized controlled trials (RCTs) |
The Cochrane Risk of Bias 2.0 tool asks questions about five types of potential bias for individually randomized trials:
|
|
|
Non-randomized studies |
The Newcastle-Ottawa scale assesses the quality of nonrandomized studies based on three broad perspectives:
|
| CASP Appraisal Checklists | Mixed methods |
These quality assessment checklists ask 11 or 12 questions each to help you identify
Available study designs include randomized controlled trials, systematic reviews, qualitative studies, cohort studies, diagnostic studies, case control studies, economic evaluations, and clinical prediction rules. |
| LEGEND Evidence Evaluation Tools | Mixed methods |
These evidence evaluation tools ask questions each to help you examine
across the clinical question domains of intervention, diagnosis & assessment, prognosis, etiology & risk factors, incidence, prevalence, and meaning. Available study designs include systematic review / meta analysis, meta-synthesis, randomized controlled trials, controlled clinical trials, psychometric studies, cohort-prospective / retrospective, case control, longitudinal, cross sectional, descriptive / epidemiology / case series, qualitative study, quality improvement, mixed methods, decision analysis / economic analysis / computer simulation, case report / n-of-1 study, published expert opinion, bench studies, and guidelines. |
Use Covidence for quality assessment
Covidence uses Cochrane Risk of Bias (which is designed for rating RCTs and cannot be used for other study types) as the default tool for quality assessment of included studies. You can opt to manually customize the quality assessment template and use a different tool better suited to your review. More information about quality assessment using Covidence, including how to customize the quality assessment template, can be found below. If you decide to customize the quality assessment template, you cannot switch back to using the Cochrane Risk of Bias template.
More Information
- Covidence FAQs on Quality AssessmentCommonly asked questions about quality assessment using Covidence
- Covidence YouTube ChannelA collection of Covidence-created videos