Systematic Reviews- Legacy Guide
Systematic review steps
During the course of conducting your systematic review, you should go through each of the following steps:
- Developing the topic and refining your PICO
When you begin your systematic review, you will want to think about the population, interventions, comparators, and outcomes that you are interested in examining. You can then use these criteria to help develop your Key Question(s), which will serve as the overall guiding force behind your systematic review. To learn more about developing your PICO and Key Question, check out the HSL Systematic Review Intake Form.
- Determining your study eligibility criteria
In addition to your PICO, you will want to determine which other criteria you want each included study to have, or which criteria will make a study ineligible for your review. A few criteria to consider include study design, dates of interest, whether you will include non-English studies, study duration, and setting.
- Searching for studies
In this step, you will design a comprehensive search strategy to capture the majority of the literature on your given topic. You can use your PICO as the starting point when brainstorming for terms to search. Once you have created your initial search, you will also want to decide which databases (e.g. PubMed, Embase, Cochrane, etc.) to search and translate your search for those databases. In addition to the published literature, you will also want to consider searching for unpublished or "grey" literature. Grey literature can be found in many sources, including manufacturer's websites, the FDA website for drug information, and clinical trial registries like ClinicalTrials.gov and the WHO International Clinical Trials Registry Platform.
- Creating and registering your protocol
Your protocol should contain all of the steps outlined above and a plan for all of the steps outlined below. You can register your protocol at PROSPERO, to ensure that someone else starting a review will not be duplicating your efforts.
- Screening and selecting studies
This step includes both a preliminary review of all titles/abstracts to determine which studies are potentially eligible, and an additional, more in-depth review of the full text for all includes from the title/abstract phase to determine final eligibility for the review. This step will require two people to review each abstract and each full-text document, plus an additional reviewer to serve as a tie breaker.
- Data extraction
You can use your PICO as a guide for the data you want to extract, including information on the study population, intervention or exposure, comparator (if applicable), outcomes data, timing of data, and study design. There are a number of tools that you can use for this step, ranging from a basic Excel setup to more sophisticated tools designed for systematic review data abstraction, like AHRQ's Systematic Review Data Repository (SRDR) and Distiller SR.To complete this step, you will need at least one team member to go through and extract all relevant data from each included study and one team member to review the data they have extracted.
- Quality assessment
In this step, you will assess the internal validity of each study, including systematic errors (e.g. selection bias, confounding), nonsystematic errors (errors attributable to chance), and inferential errors (problems in data analysis and interpretation). Depending on the types of studies you are examining, you may want to select a pre-existing quality rating tool for each study type. A sample of quality rating tools can be found in the box below.
- Assessment of applicability
In this step, you will assess the external validity of each study (how well the results of the study are likely to reflect real-world outcomes), as well as the external validity of the body of literatures as a whole. You can also use your PICO to determine the most important factors affecting applicability. Some issues to consider here are how narrow the exclusion criteria were within each study, differences between patients in the study and in the community, whether or not a medication regimen is reflective of current practice, and whether an intervention's intensity would be feasible for routine use.
- Presentation of findings
In this step, you will develop your evidence tables, which give detailed information for each study (using your PICO as a guide), and summary tables, which give a high level overview of the findings of your review. You can create evidence and summary tables to describe study characteristics, results, or both. These tables will help you determine which studies, if any, are eligible for quantitative synthesis.
- Quantitative synthesis
Quantitative synthesis, also known as meta-analysis or pooling, is the process of combining data from multiple studies in order to gain a sense of the overall effect of a treatment or exposure. When determining whether or not a meta-analysis should be undertaken, great care should be taken to determine whether or not the studies are similar enough to analyze together, and whether or not an analysis will provide potentially misleading results. There are a number of tools that can be used to conduct a meta-analysis, including RevMan, CMA, and Stata.
- Grading the strength of evidence
Strength of evidence grades are used to evaluate the strength of the entire body of evidence for major outcomes and comparisons. Strength of evidence is a single grade based on four domains: risk of bias, consistency, directness, and precision. For more information on grading strength of evidence, check out the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group.
- Writing your review
Your final review manuscript should contain the following pieces: title, abstract, introduction, methods, results, discussion, and disclosure of funding. For more information on the information contained in each section, see the full descriptions in "Systematic Reviews - Parts and Pieces" below. In addition to these main pieces, you should also consider include appendices with your complete evidence tables and quality assessment forms and a line-by-line description of your search methodology.
For additional, step-by-step guidance, check out the AHRQ Training Modules, or one of the other online training options below.
Systematic reviews - parts and pieces
Your systematic review should contain each of the following sections:
- TITLE - The title should identify what the report is about and what it is (systematic review, meta-analysis, or both).
- ABSTRACT - The abstract should provide a structured summary of the parts of the review.
- INTRODUCTION - The introduction should describe why the review is needed (the rationale for the review) and the review objectives (the question being addressed).
- METHODS - The methods section should include a thorough description of how the systematic review was conducted, including whether or not you created a protocol and where it is registered, what your eligibility or inclusion criteria were, what your sources of information were and the date they were searched, your full search strategy, your process for selecting studies and collecting data, the data collected, how risk of bias was assessed on an individual study basis and across studies, your summary measures (risk ratio, difference in mean, etc.), and how results were synthesized and analyzed. If you have a protocol, much of this information can be taken from there.
- RESULTS - The results section should include the findings of your review, including a description of the studies screened for eligibility and the studies selected, the characteristics of the included studies, the risk of bias for each individual study, the results of each individual study, a synthesis of the data for each outcome considered, the risk of bias across studies, and any additional analysis conducted.
- DISCUSSION - The discussion section should include a summary of the main findings of the report, including the strength of evidence for each outcome and the findings' relevance to key groups, the limitations of the report, and a conclusion that contains a general interpretation of the results.
- FUNDING - Any funding received for the review and the role of the funders in the review should be described in this section.
For more information, check out the PRISMA checklist.