Evidence Synthesis: Step 5: Screening

After you have retrieved and deduplicated your citations, you will upload them into a screening tool to assess the relevance of each citation. "Screening" refers to the process of deciding whether a citation or document is relevant to the review. The inclusion & exclusion criteria previously outlined by the team will be used in this step to determine relevance.

Process

1. At least two people vote on the relevance of every citation by reading the titles & abstracts. 

2. For all citations identified as potentially relevant, the team must retrieve the full text documents.

3. At least two people vote on the relevance of every document by reading the full text.

People

It is recommended that you have at least two to three people doing screening, although more people can be involved. Two people should screen each citation independently and should not be able to see how the other screener is voting. Where there are disagreements, the third person will be the tiebreaker. At least one screener should be a content matter expert. 

When considering how many team members should be involved in screening, keep in mind:

• The fewer screeners you have, the more citations each screener will need to review.
• The more screeners you have, the more effort will be required to ensure each screener has the same understanding of the research question and inclusion criteria.

Tools

UNC affiliates have full access to Covidence through our institutional subscription. There are several other options available, but please note that UNC Libraries only provide support for Covidence.

Benefits of Using Covidence

• Blinded screening
• Automatic highlighting for key words that indicate inclusion/exclusion criteria
• Machine learning (optional feature) ranks citations by relevance
• Creates a PRISMA diagram

Other tools

• Rayyan
• EPPI-Reviewer
• Abstraktr
• Excel and Citation Managers: These tools are no longer widely used for screening due to their limitations. While you may opt to use them, it's recommended that UNC affiliates make use of the institutional subscription to Covidence.
• To learn more about software tools for screening see the peer-reviewed article by Harrison et al. (2020).

Title & Abstract Screening

Prior to beginning the full title/abstract screening stage, the Cochrane Handbook recommends doing a test run with a random small subset of their citations. This pilot screening step allows the team to make sure everyone is on the same page regarding key definitions, inclusion criteria, and reasons for exclusion. A pilot screening can be set up and run in Covidence. 

During the title/abstract screening stage, it is best to err on the side of caution when in doubt and include citations for full text screening. 

Screeners do not need to identify the reason why a citation is excluded at this stage. The total number of citations excluded at this stage, however, must be reported. 

Full Text Retrieval

After title/abstract screening, you'll need to retrieve the full text PDFs for each citation that has moved up to the next stage of screening. Every citation at the full text stage must have a full text PDF to review. The steps for this piece include:

1. Export all citations from Covidence that have progressed through the title/abstract stage up to the full text stage.

2. Import the citations into a citation manager (EndNote, Zotero).

3. Use the citation manager to automatically retrieve as many PDFs as possible.

4. Manually retrieve PDFs for any citations that are still missing full text. 

5. Export the citations from the citation manager.

6. Import the PDFs into Covidence using the Bulk Upload feature.

Details for all of these steps can be found in the HSL research guide page Adding Full Text into your Covidence review. 

Full Text Screening

After retrieving the full text PDFs for all citations that have advanced to this stage, two screeners will review the full texts independently. When a full text is excluded at this stage, screeners do need to identify the reason why it is excluded. Although it may be excluded for multiple reasons, the Cochrane Handbook notes that one reason suffices. The reasons for exclusion, and the number of full texts that fall under each reason, must be reported.

Screening, data extraction, and QA steps can be undertaken simultaneously in larger teams that split these tasks up. For example, as citations are deemed relevant in the title/abstract screening stage, other team members can begin retrieving the full text documents and screening in the full text stage.

Tips for manually finding full text:

1. For book chapters, search for the book title rather than the chapter.

2. Conference abstracts typically are not indexed well on journal websites and are therefore difficult to search for by title. Instead, try navigating to the volume/issue indicated in the citation and scrolling to the page listed.

3. Many articles and dissertations are deposited in institutional repositories. A Google search for the title will produce results from IRs and you should be able to grab the full text if it has been deposited.

4. For all other materials you can’t locate, just fill out an ILL request form with as much information about the citation as possible and our team will look for the item from another library.

Waffenschmidt, S., Knelangen, M., Sieben, W., Buhn, S., & Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology, 19(1), 132. 10.1186/s12874-019-0782-0 Find@UNC

Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020). Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Medical Research Methodology, 20, 1-12. 10.1186/s12874-020-0897-3 Find@UNC

Mitchell, N., & Ennis, L. A. (2022). Evaluating for Inclusion. In M. J. Foster & S. T. Jewell (Eds.), Piecing Together Systematic Reviews and Other Evidence Syntheses (pp. 173-181). Find@UNC

Lefebvre C., Glanville J., Briscoe S., Littlewood A., Marshall C., Metzendorf M.-I., Noel-Storr A., Rader T., Shokraneh F., Thomas J., Wieland L.S. Chapter 4: Searching for and selecting studies. In: Higgins J.P.T., Thomas J., Chandler J., Cumpston M., Li T., Page M.J., Welch V.A. (editors). Cochrane Handbook for Systematic Reviews of Interventions. 2nd Edition. Chichester (UK): John Wiley & Sons, 2019: 67–108. Link to full text

MacDonald, H., Comer, C., Foster, M., Labelle, P. R., Marsalis, S., Nyhan, K., Premji, Z., Rogers, M., Splenda, R., Stansfield, C., & Young, S. (2024). Searching for studies: A guide to information retrieval for Campbell systematic reviews. Campbell Systematic Reviews, 20(3), e1433. 10.1002/cl2.1433 Link to full text