Skip to Main Content

Scoping Reviews: Step 6: Assess Quality of Included Studies (Optional)

Created by Health Science Librarians

Optional Step Information

Stop sign

The quality assessment step is one of the main differences in systematic and scoping review methodology.
While assessing quality of included studies is a required step of systematic reviews, it is optional for scoping reviews.

About Step 6: Assess Quality of Included Studies

It takes an average of 68 hours for a systematic review team to perform quality assessment on included studies

Click an item below to see how it applies to Step 6: Assess Quality of Included Studies.

If you reach the quality assessment step and choose to exclude articles for any reason, update the number of included and excluded studies in your PRISMA flow diagram.

Covidence includes the Cochrane Risk of Bias 2.0 quality assessment template, but you can also create your own custom quality assessment template.

A librarian can advise you on quality assessment for your scoping review, including: 

  • What the quality assessment or risk of bias stage of the review entails
  • How to choose an appropriate quality assessment tool
  • Best practices for reporting quality assessment results in your review

 

Assess studies for quality and bias

After the screening process is complete, the scoping review team may assess each article for quality and bias. There are various types of bias, some of which are outlined in the table below from the Cochrane Handbook.

The most important thing to remember when choosing a quality assessment tool is to pick one that was created and validated to assess the study design(s) of your included articles.

For example, if one item in the inclusion criteria of your scoping review is to only include randomized controlled trials (RCTs), then you need to pick a quality assessment tool specifically designed for RCTs (for example, the Cochrane Risk of Bias tool)

Critically appraise included studies

Once you have gathered your included studies, you will need to appraise the evidence for its relevance, reliability, validity, and applicability​.

Ask questions like:When you appraise, you should evaluate for relevance, reliability, validity, and applicability

Relevance:  ​

  • Is the research method/study design appropriate for answering the research question?​
  • Are specific inclusion / exclusion criteria used? ​

Reliability:  ​

  • Is the effect size practically relevant? How precise is the estimate of the effect? Were confidence intervals given?  ​

Validity: ​

  • Were there enough subjects in the study to establish that the findings did not occur by chance?    ​
  • Were subjects randomly allocated? Were the groups comparable? If not, could this have introduced bias?  ​
  • Are the measurements/ tools validated by other studies?  ​
  • Could there be confounding factors?   ​

Applicability:  ​

  • Can the results be applied to my organization and my patient?   ​

Select a quality assessment tool

What are Quality Assessment tools?

Quality Assessment tools are questionnaires created to help you assess the quality of a variety of study designs.  Depending on the types of studies you are analyzing, the questionnaire will be tailored to ask specific questions about the methodology of the study.  There are appraisal tools for most kinds of study designs.  You should choose a Quality Assessment tool that matches the types of studies you expect to see in your results.  If you have multiple types of study designs, you may wish to use several tools from one organization, such as the CASP or LEGEND tools, as they have a range of assessment tools for many study designs.

Click on a study design below to see some examples of quality assessment tools for that type of study.

 

 

 

  • STROBE Checklist
    A checklist for quality assessment of case-control, cohort, and cross-sectional studies
  • LEGEND Evidence Evaluation Tools
    A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.

 

 

 

 

  • CASP- Qualitative Studies
    10 questions to help assess qualitative research from the Critical Appraisal Skills Programme
  • LEGEND Evidence Evaluation Tools
    A series of critical appraisal tools from the Cincinnati Children's Hospital. Contains tools for a wide variety of study designs, including prospective, retrospective, qualitative, and quantitative designs.

 

 

 

  • NTACT Quality Checklists
    Quality indicator checklists for correlational studies, group experimental studies, single case research studies, and qualitative studies developed by the National Technical Assistance Center on Transition (NTACT). (Users must make an account.)

 

A closer look at popular tools

Below, you will find a sample of four popular quality assessment tools and some basic information about each. For more quality assessment tools, please view the blue tabs in the boxes above, organized by study design.

More information about popular quality assessment tools
Tool Study Design About
Cochrane Risk of Bias 2.0 Tool Randomized controlled trials (RCTs)

The Cochrane Risk of Bias 2.0 tool asks questions about five domains of potential bias for individually randomized trials:

  • bias arising from the randomization process
  • bias due to deviations from intended interventions
  • bias due to missing outcome data
  • bias in measurement of the outcome
  • bias in selection of the reported result

Newcastle-Ottawa Scale

 Non-randomized studies

The Newcastle-Ottawa scale assesses the quality of nonrandomized studies based on three broad perspectives:

  • the selection of the study groups
  • the comparability of the groups
  • the ascertainment of either the exposure or outcome of interest for case-control or cohort studies respectively
CASP Appraisal Checklists Mixed methods

These quality assessment checklists ask 11 or 12 questions each to help you identify

  • Is the basic study design valid?
  • Was the study methodologically sound?
  • What are the results?
  • Will the results help locally?

Available study designs include randomized controlled trials, systematic reviews, qualitative studies, cohort studies, diagnostic studies, case control studies, economic evaluations, and clinical prediction rules.
LEGEND Evidence Evaluation Tools Mixed methods

These evidence evaluation tools ask questions each to help you examine

  • validity
  • reliability
  • applicability

across the clinical question domains of intervention, diagnosis & assessment, prognosis, etiology & risk factors, incidence, prevalence, and meaning.

Available study designs include systematic review / meta analysis, meta-synthesis, randomized controlled trials, controlled clinical trials, psychometric studies, cohort-prospective / retrospective, case control, longitudinal, cross sectional, descriptive / epidemiology / case series, qualitative study, quality improvement, mixed methods, decision analysis / economic analysis / computer simulation, case report / n-of-1 study, published expert opinion, bench studies, and guidelines.

Use Covidence for quality assessment

Covidence uses Cochrane Risk of Bias (which is designed for rating RCTs and cannot be used for other study types) as the default tool for quality assessment of included studies. You can opt to manually customize the quality assessment template and use a different tool better suited to your review. More information about quality assessment using Covidence, including how to customize the quality assessment template, can be found below. If you decide to customize the quality assessment template, you cannot switch back to using the Cochrane Risk of Bias template.

More Information

  • Last Updated: Sep 25, 2024 4:01 PM
  • URL: https://guides.lib.unc.edu/scoping-reviews