Evidence Synthesis: Data Extraction and Quality Assessment

After completing the full text screening, you will have your final list of included sources. The next step is to extract data from each source. If you are doing a systematic review, you'll do a quality assessment (also called a critical appraisal or risk of bias assessment) for each source as well. 

"Data" can refer to information about the citation itself (e.g., title, year) as well as information contained within the citation (e.g., participant demographics). If you aren't sure what data should be extracted, the Libraries' Data Services team can help you identify what data you'll want to collect in order to answer your research question.

The data you extract should be outlined in your protocol. It's okay to deviate from what is written in your protocol; just provide an explanation for these changes in your final manuscript.

Two team members should separately review each citation and extract data. Duplication of this work is meant to ensure accuracy. In cases where the extracted data does not match, a third team member may review and decide. 

Two members of the team will independently perform quality assessments for each publication that has passed the full text screening stage. Similar to during screening, this process should be blinded so the reviewers do not see each other's work.

If you have a large team, you may choose to have more than two people perform quality assessments. This will speed up the process, but it also requires training and oversight to make sure that everyone performing QA is on the same page. 

There are several tools available for data extraction. Covidence includes a customizable data extraction tool. Visit HSL's Covidence research guide to learn more about using Covidence for data extraction.  

Tips for Data Extraction

• Be careful about extracting more data than you need for your review. The more data you extract, the more time and energy your team needs to invest in the extraction phase.
• Do a test run to make sure all team member who will be extracting data are on the same page about what data to extract and how to extract it.

Quality assessment (also called risk of bias or critical appraisal) is necessary for systematic reviews. If you are conducting a scoping review, this part is optional. 

The quality assessment step is important for identifying biases, conflicts of interest, and insufficient methodological approaches. It is especially critical for systematic reviews because these aim to reach a consensus on a specific topic based on the best available evidence. Determining the best available evidence -- and if a consensus can be reached at all -- happens during this step.

People

Two team members will independently perform quality assessments for each publication that has passed the full text screening stage. Similar to during screening, this process should be blinded so the reviewers do not see each other's work.  In cases where the assessments do not match, a third team member may review and decide. 

If you have a large team, you may choose to have more than two people perform quality assessments. This will speed up the process, but it also requires training and oversight to make sure that everyone performing QA is on the same page. 

Quality Assessment Tools

The team should identify a QA tool during the protocol stage. A QA tool will provide a checklist of items to assess when reviewing each publication. 

A list of QA tools is available on the HSL Guide here. 

Tips for Quality Assessment

• Every publication is likely to have bias or small errors. The key part of appraisal is being able to differentiate between small issues that have little impact on the findings, and large issues that may have major impacts on the study's overall quality.
• Keep in mind that many QA tools were designed for appraising health sciences research that is largely quantitative in nature and is conducted through randomized controlled trials (RCTs). Social sciences research relies more on qualitative and mixed methods research.
  • Aspects of "high quality" research in the health sciences, such as double blinding and having a control group, may not be feasible in social sciences research, and therefore may not be a good indication of quality in your discipline.
  • It is more important to choose a tool that was designed to fit the studies in your review, rather than choosing a tool that is widely used but will not provide an accurate assessment of your included studies. 

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies [last updated August 2022]. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane, 2024. Available from www.training.cochrane.org/handbook.

Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences : a practical guide. Blackwell Pub. 9781405150149 Find@UNC