Systematic Review Automation: Learn about SR Automation

Systematic reviews (SRs) require a lot of time and effort.  The SR process can take over a year to complete and reviews are often outdated at the time of publication.  Furthermore, Cochrane states that reviews should be considered for updates every 2 years.

The technology itself has been around for many years, and automation tools have been used in systematic reviews for over 10 years.  Use of these tools has not yet reached the majority of those performing systematic reviews.

• Allen, I. E., & Olkin, I. (1999). Estimating time to conduct a meta-analysis from number of citations retrieved. Jama, 282(7), 634-635.
• Bastian, H., Glasziou, P., & Chalmers, I. (2010). Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Medicine, 7(9), e1000326.
• Beller, E. M., Chen, J. K., Wang, U. L., & Glasziou, P. P. (2013). Are systematic reviews up-to-date at the time of publication? Systematic reviews, 2, 36. doi:10.1186/2046-4053-2-36
• Chapman, A. L., Morgan, L. C., & Gartlehner, G. (2010). Semi-automating the manual literature search for systematic reviews increases efficiency. Health Information & Libraries Journal, 27(1), 22-27. doi:10.1111/j.1471-1842.2009.00865.x
• Haddaway, N. R., & Westgate, M. J. (2018). Predicting the time needed for environmental systematic reviews and systematic maps. Conserv Biol. doi:10.1111/cobi.13231
• Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016). Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic reviews, 5(1), 140. doi:10.1186/s13643-016-0315-4
• Stansfield, C., O'Mara-Eves, A., & Thomas, J. (2017). Text mining for search term development in systematic reviewing: A discussion of some methods and challenges. Research Synthesis Methods, 8(3), 355-365. doi:10.1002/jrsm.1250
• Stansfield, C. M., O'Mara-Eves, A., & Thomas, J. (2015). Reducing systematic review workload using text mining: opportunities and pitfalls. Journal of the European Association for Health Information and Libraries, 11(3), 8-10.
• Thomas, J. (2013). Diffusion of innovation in systematic review methodology: why is study selection not yet assisted by automation. OA Evidence-Based Medicine, 1(2), 1-6.
• Tsafnat, G., Dunn, A., Glasziou, P., & Coiera, E. (2013). The automation of systematic reviews. BMJ (Clinical Research Ed.), 346, f139. doi:10.1136/bmj.f139
• van Altena, A. J., Spijker, R., & Olabarriaga, S. D. (2019). Usage of automation tools in systematic reviews. Research synthesis methods, 10(1), 72-82. doi:10.1002/jrsm.1335

Automation can help speed up the process for researchers, students, and librarians through many stages of the review, including searching, deduplication, screening, bias assessment, and extraction.

• Adeva, J. G., Atxa, J. P., Carrillo, M. U., & Zengotitabengoa, E. A. (2014). Automatic text classification to support systematic reviews in medicine. Expert Systems with Applications, 41(4), 1498-1508.
• Cunningham, S. J., Witten, I. H., Litten, J., & Williams, M. E. (2001). Applications of machine learning in information retrieval. In (pp. 341-384).
• Garc, J. J., Adeva, a., Atxa, J. M. P., Carrillo, M. U., & Zengotitabengoa, E. A. (2014). Automatic text classification to support systematic reviews in medicine. None, 41, 1498-1508. doi:10.1016/j.eswa.2013.08.047
• Hemens, B. J., & Iorio, A. (2017). Computer-aided systematic review screening comes of age. Ann Intern Med, 167, 210-211. doi:10.7326/M17-1295
• Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015). Automating data extraction in systematic reviews: a systematic review. Systematic reviews, 4, 78. doi:10.1186/s13643-015-0066-7
• O'Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015). Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic reviews, 4, 5. doi:10.1186/2046-4053-4-5
• Thomas, J., McNaught, J., & Ananiadou, S. (2011). Applications of text mining within systematic reviews. Research Synthesis Methods, 2(1), 1-14.
• Waffenschmidt, S., Hausner, E., Sieben, W., Jaschinski, T., Knelangen, M., & Overesch, I. (2018). Effective study selection using text mining or a single-screening approach: a study protocol. Systematic reviews, 7, 166. doi:10.1186/s13643-018-0839-x

“Living systematic review” (LSR): systematic reviews that are continually updated, incorporating relevant new evidence as it becomes available. LSRs may be particularly important in fields where research evidence is emerging rapidly, current evidence is uncertain, and new research may change policy or practice decisions.

Living systematic reviews

(Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., . . . Thomas, J. (2017). Living systematic review: 1. Introduction-the why, what, when, and how. Journal of Clinical Epidemiology, 91, 23-30. doi:10.1016/j.jclinepi.2017.08.010)

• Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014). Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Medicine, 11(2), e1001603. doi:10.1371/journal.pmed.1001603
• Akl, E. A., Meerpohl, J. J., Elliott, J., Kahale, L. A., Schünemann, H. J., Agoritsas, T., . . . Pearson, L. (2017). Living systematic reviews: 4. Living guideline recommendations. Journal of Clinical Epidemiology, 91, 47-53. doi:https://doi.org/10.1016/j.jclinepi.2017.08.009
• Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., . . . Thomas, J. (2017). Living systematic review: 1. Introduction-the why, what, when, and how. Journal of Clinical Epidemiology, 91, 23-30. doi:10.1016/j.jclinepi.2017.08.010
• Simmonds, M., Salanti, G., McKenzie, J., Elliott, J., Agoritsas, T., Hilton, J., . . . Pearson, L. (2017). Living systematic reviews: 3. Statistical methods for updating meta-analyses. Journal of Clinical Epidemiology, 91, 38-46. doi:https://doi.org/10.1016/j.jclinepi.2017.08.008
• Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., . . . Elliott, J. (2017). Living systematic reviews: 2. Combining human and machine effort. Journal of Clinical Epidemiology, 91, 31-37. doi:10.1016/j.jclinepi.2017.08.011

• Enck, P. (2018). Living systematic reviews, not only for clinical (placebo) research. Journal of Clinical Epidemiology, 98, 152-153. doi:https://doi.org/10.1016/j.jclinepi.2018.01.001

Many of these tools are included in the Systematic Review Toolbox at systematicreviewtools.com

You can search by part of review or AI approach and see a list of tools and accompanying articles.

• Chris Marshall, B. K., Pearl Brereton. (2018). Tool Features to Support Systematic Reviews in Software Engineering – A Cross Domain Study. e-Informatica Software Engineering Journal, 12(1), 79–115. doi:10.5277/e-Inf180104
• Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018). Online tools supporting the conduct and reporting of systematic reviews and systematic maps: a case study on CADIMA and review of existing tools. Environmental Evidence, 7(1), 8. doi:10.1186/s13750-018-0115-5
• Marshall, C. (2016). Tool support for systematic reviews in software engineering (Doctor of Philosophy), Keele University, Retrieved from http://eprints.keele.ac.uk/2431/1/MarshallPhD2016.pdf
• Marshall, C., Brereton, P., & Kitchenham, B. (2014). Tools to support systematic reviews in software engineering: a feature analysis. Paper presented at the Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, London, England, United Kingdom. https://dl.acm.org/citation.cfm?doid=2601248.2601270
• Marshall, C., Brereton, P., & Kitchenham, B. (2015). Tools to support systematic reviews in software engineering: a cross-domain survey using semi-structured interviews. None
• Park, S. E., & Thomas, J. (2018). Evidence synthesis software. Evidence-Based Medicine. doi:10.1136/bmjebm-2018-110962
• Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014). Systematic review automation technologies. Systematic reviews, 3, 74. doi:10.1186/2046-4053-3-74

• Edwards, M., & Marshall, C. (2017). Evaluating Robotreviewer for Automated Risk of Bias Assessment in a Systematic Review: A Case Study. Value in Health, 20(9), A774.
• Fabbri, S., Silva, C., Hernandes, E., Octaviano, b., Andr, Thommazo, D., & Belgamo, A. (2016). Improvements in the StArt tool to better support the systematic review process. None
• Gates, A., Johnson, C., & Hartling, L. (2018). Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Systematic reviews, 7, 45. doi:10.1186/s13643-018-0707-8 10.1186/s13643-018-0707-8.']
• Gates, A., Vandermeer, B., & Hartling, L. (2018). Technology-assisted risk of bias assessment in systematic reviews: a prospective cross-sectional evaluation of the RobotReviewer machine learning tool. Journal of Clinical Epidemiology, 96, 54-62. doi:10.1016/j.jclinepi.2017.12.015
• Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., . . . Thayer, K. (2016). SWIFT-Review: a text-mining workbench for systematic review. Systematic reviews, 5, 87. doi:10.1186/s13643-016-0263-z
• Johnson, N., & Phillips, M. (2018). Rayyan for systematic reviews. Journal of Electronic Resources Librarianship, 30, 46-48. doi:10.1080/1941126X.2018.1444339
• Marshall, C., & Brereton, P. (2015). Systematic review toolbox: a catalogue of tools to support systematic reviews. Paper presented at the Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, Nanjing, China. https://dl.acm.org/citation.cfm?doid=2745802.2745824
• Marshall, I. J., Jo, Kuiper, l., & Wallace, B. C. (2014). Automating risk of bias assessment for clinical trials. None
• Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016). RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Association, 23(1), 193-201. doi:10.1093/jamia/ocv044
• Nur, S., Adams, C. E., & Brailsford, D. F. (2016). Using built-in functions of Adobe Acrobat Pro DC to help the selection process in systematic reviews of randomised trials. Systematic reviews, 5(1). doi:10.1186/s13643-016-0207-7
• Olofsson, H., Brolund, A., Hellberg, C., Silverstein, R., Stenström, K., Österberg, M., & Dagerhamn, J. (2017). Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan. Research synthesis methods, 8, 275-280. doi:10.1002/jrsm.1237
• Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan-a web and mobile app for systematic reviews. Systematic reviews, 5(1), 210. doi:10.1186/s13643-016-0384-4
• Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M. A., McNaught, J., von Elm, E., . . . Ananiadou, S. (2018). Prioritising references for systematic reviews with RobotAnalyst: a user study. Research Synthesis Methods.
• Rathbone, J., Hoffmann, T., & Glasziou, P. (2015). Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Systematic reviews, 4, 80. doi:10.1186/s13643-015-0067-6
• Saldanha, I. J., Schmid, C. H., Lau, J., Dickersin, K., Berlin, J. A., Jap, J., . . . Li, T. (2016). Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Systematic reviews, 5(1), 196. doi:10.1186/s13643-016-0373-7
• Tan, M. C. (2018). PRODUCT REVIEW: Colandr. Journal of the Canadian Health Libraries Association (JCHLA), 39, 85-88. doi:10.29173/jchla29369
• Thomas, J., Brunton, J., & Graziosi, S. (2010). EPPI-Reviewer 4.0: software for research synthesis. EPPI-Centre Software. London: Social Science Research Unit, Institute of Education.
• Torres Torres, M., & Adams, C. E. (2017). RevManHAL: towards automatic text generation in systematic reviews. Systematic reviews, 6(1), 27. doi:10.1186/s13643-017-0421-y
• Von Philipsborn, P., Busert, L., Burns, J., Pfadenhauer, L. M., Polus, S., Stratil, J. M., & Rehfuess, E. A. (2016). Using specialized software to streamline the production of systematic reviews of complex public health interventions: A comparison of Covidence and abstrackr. Eur J Epidemiol, 31, S221-S222. doi:10.1007/s10654-016-0183-1
• Waffenschmidt, S., Hausner, E., Sieben, W., Jaschinski, T., Knelangen, M., & Overesch, I. (2018). Effective study selection using text mining or a single-screening approach: a study protocol. Systematic reviews, 7, 166. doi:10.1186/s13643-018-0839-x 10.1186/s13643-018-0839-x.']
• Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, TA. (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the 2nd ACM SIGHIT Symposium on International Health Informatics. doi:https://doi.org/10.1145/2110363.2110464
• Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2010). Modeling annotation time to reduce workload in comparative effectiveness reviews. None
• Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. None
• Westgate, M. J. (2018). revtools: bibliographic data visualization for evidence synthesis in R. BioRxiv. doi:10.1101/262881
• Westgate, M. J., Haddaway, N. R., Cheng, S. H., McIntosh, E. J., Marshall, C., & Lindenmayer, D. B. (2018). Software support for environmental evidence synthesis. Nature Ecology & Evolution, 2(4), 588-590. doi:10.1038/s41559-018-0502-x
• Yang, J. J., Cohen, A. M., & McDonagh, M. S. (2008). SYRIAC: The systematic review information automated collection system a data warehouse for facilitating automated biomedical text classification. AMIA Annual Symposium Proceedings, 2008, 825.
• Yu, W., Clyne, M., Dolan, S. M., Yesupriya, A., Wulf, A., Liu, T., . . . Gwinn, M. (2008). GAPscreener: an automatic tool for screening human genetic association literature in PubMed using the support vector machine technique. BMC bioinformatics, 9, 205. doi:10.1186/1471-2105-9-205

• Hempel, S., Shetty, K. D., Shekelle, P. G., Rubenstein, L. V., Danz, M. S., Johnsen, B., & Dalal, S. R. (2012). Machine learning methods in systematic reviews: identifying quality improvement intervention evaluations.
• Paynter, R., Banez, L. L., Berliner, E., Erinoff, E., Lege-Matsuura, J., Potter, S., & Uhl, S. (2016). EPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews Retrieved from Rockville (MD):