why do systematic literature review

Unfortunately we don't fully support your browser. If you have the option to, please upgrade to a newer version or use Mozilla Firefox , Microsoft Edge , Google Chrome , or Safari 14 or newer. If you are unable to, and need support, please send us your feedback .

We'd appreciate your feedback. Tell us what you think! opens in new tab/window

Why systematic reviews matter

July 23, 2019 | 7 min read

By Tina Poklepović Peričić, Sarah Tanveer

A brief history, overview and practical guide for authors

This article was written as part of the Cochrane International Mobility Programme.

Introduction

The vast scale of scholarly literature occasions various problems. One is how to comprehensively record and assess the state of knowledge on a particular topic. A potent means of doing so is that of systematic reviews. The continuous growth of research, coupled with the demand to systematically summarize the available evidence to inform decisions from consumers and stakeholders, led to the formal development of systematic reviews (SRs) in the late 20th century 1-3 .

Systematic reviews search, appraise and collate all relevant empirical evidence in order to provide a complete interpretation of research results. Although conventional SRs are typically used in clinical research and social sciences, they have found application in various other subject areas for example in advertising, education, international development, public policy, ecology, environmental sciences, engineering and basic science research 4-7

A brief history of systematic reviews

The first example of a systematic review was conducted in 1753 by James Lind, who published a paper that aimed to provide a concise and unbiased summary of evidence on scurvy 8 9 . However, it was not until the 1970’s and 1980’s when more attention was paid to the growing need to improve the state of evidence synthesis.

In 1972, Archie Cochrane published a textbook titled “ Effectiveness and Efficiency: Random Reflections on Health Service ” 10 . Cochrane drew attention to the vital importance of randomized control trials in determining the effectiveness of health treatments. This led to a greater international emphasis on the need to improve research synthesis by policy makers, academics, and clinicians 3 . Gradually, topic areas outside of healthcare also adopted SRs as a way of comprehensively and systemically summarizing existing research.

How to conduct a systematic review

If you are considering embarking on a systematic review, there are several issues you need to contemplate if you wish to conduct one. In healthcare, for example, the first step would be to define an explicit research question by using the PICOTS (Population, Intervention, Comparator, Outcome, Timing, Setting) framework 11 , and also register the protocol for the review on PROSPERO ( https://www.crd.york.ac.uk/prospero/ opens in new tab/window ), the international database of prospectively registered systematic reviews. Protocols provide a complete detailed description of the process by which the review will be conducted. Registering the protocol reduces research bias, duplication of effort, resource waste, and provides greater transparency 12 . Outside of medical sciences, protocols can be uploaded to Open Science Framework ( https://osf.io opens in new tab/window ).

You must adopt a comprehensive, objective and reproducible search strategy to capture all relevant sources of evidence. In doing so, you can be confident of having incorporated all the appropriate material for the topic at hand. A thorough search strategy should involve multiple databases, registries, sources of grey literature ( https://onlinelibrary.london.ac.uk/resources/databases/opengrey opens in new tab/window ) 13 , conference proceedings and abstracts. Following the predefined eligibility criteria, you then need to analyze the screened search results to extract data from those publications that meet the inclusion criteria.

Don’t forget to assess the risk of bias when applicable (i.e., in clinical research). Ideally, these methodological steps should preferably be performed by two authors independently, one of which is a methodologist and the other a content area expert. Summarizing the results of the included studies and interpreting their findings in the light of certainty of evidence and their applicability are the final steps of completing a systematic review. You can also include a meta-analysis if applicable.

In order to assess the methodological quality of systematic reviews in biomedical sciences, checklists like the AMSTAR – “A MeaSurement Tool to Assess systematic Reviews" 14 (https://amstar.ca) can be utilized. Lastly, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 16 (https://prisma-statement.org/) checklist, is a minimum set of items for reporting in systematic reviews: include one with your full article. The flow diagram can also be adopted to use for non-medical research (https://prisma-statement.org/PRISMAStatement/FlowDiagram).

Why are systematic reviews important?

Systematic reviews offer a number of benefits. For starters, they deliver a clear and comprehensive overview of available evidence on a given topic. Moreover, SRs also help identify research gaps in our current understanding of a field. They can highlight methodological concerns in research studies that can be used to improve future work in the topic area 17 . Lastly, they can be used to identify questions for which the available evidence provide clear answers and thus for which further research is not necessary 18 .

The process of conducting systematic reviews, especially for new authors, will prove to be a worthwhile endeavour. Authors refine their knowledge on the subject area of interest, develop new research ideas, and gain critical skills in synthesising existing literature.

We hope that you have found this introduction to systematic reviews helpful. Additional information about SRs can be found on the Cochrane website opens in new tab/window . If you have any questions or observations, please feel free to comment below.

1. Meerpohl JJ, Herrle F, Reinders S, et al. Scientific value of systematic reviews: survey of editors of core clinical journals. PLoS One 2012;7(5):e35732. doi: 10.1371/journal.pone.0035732 [published Online First: 2012/05/09]

2. Higgins JPT, Green S, Cochrane Collaboration. Cochrane handbook for systematic reviews of interventions Version 5.1.0 [updated March 2011]. London: The Cochrane Collaboration,, 2011:1 online resource. doi: 10.1002/9780470712184

3. Chalmers I, Hedges LV, Cooper H. A brief history of research synthesis. Eval Health Prof 2002;25(1):12-37. doi: 10.1177/0163278702025001003 [published Online First: 2002/03/01]

4. O’Hagan EC, Matalon S, Riesenberg LA. Systematic reviews of the literature: a better way of addressing basic science controversies: American Physiological Society Bethesda, MD, 2018. doi: 10.1152/ajplung.00544.2017

5. Gilbody S, Wilson P, Watt I. Benefits and harms of direct to consumer advertising: a systematic review. BMJ Quality & Safety 2005;14(4):246-50. doi: 10.1136/qshc.2004.012781

6. Pullin AS, Stewart GB. Guidelines for systematic review in conservation and environmental management. Conservation biology 2006;20(6):1647-56. doi: 10.1111/j.1523-1739.2006.00485.x

7. Petticrew M. Systematic reviews from astronomy to zoology: myths and misconceptions. Bmj 2001;322(7278):98-101. doi: 10.1136/bmj.322.7278.98

8. Lind J. A treatise on the scurvy. In three parts. Containing an inquiry into the nature, causes, and cure, of that disease. London,: A. Millar 1753. doi:10.1136/bmj.330.7482.92-a

9. Clarke M, Chalmers I. Reflections on the history of systematic reviews. BMJ Evid Based Med 2018;23(4):121-22. doi: 10.1136/bmjebm-2018-110968 [published Online First: 2018/06/21]

10. Cochrane AL. Effectiveness and efficiency: random reflections on health services. London: Nuffield Provincial Hospitals Trust 1972. doi: 10.1017/cbo9781107256644

11. Santos CMdC, Pimenta CAdM, Nobre MRC. The PICO strategy for the research question construction and evidence search. Revista latino-americana de enfermagem 2007;15(3):508-11. doi: 10.1590/s0104-11692007000300023

12. Stewart L, Moher D, Shekelle P. Why prospective registration of systematic reviews makes sense. Syst Rev 2012;1:7. doi: 10.1186/2046-4053-1-7 [published Online First: 2012/05/17]

13. Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Research synthesis methods 2014;5(3):221-34. doi: 10.1002/jrsm.1106

14. Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj 2017;358:j4008. doi: 10.1136/bmj.j4008

15. Chandler J, Churchill R, Higgins J, et al. Methodological standards for the conduct of new Cochrane Intervention Reviews. The Cochrane Library 2013 doi: 10.4073/cpg.2016.3

16. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine 2009;151(4):264-69. doi: 10.7326/0003-4819-151-4-200908180-00135

17. Eagly AH, Wood W. Using research syntheses to plan future research. The Handbook of Research Synthesis: Russell Sage Foundation 1994:485-500. doi: 10.1002/(SICI)1097-0258(19970330)16:6<713::AID-SIM430>3.0.CO;2-4

18. Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Obstet Gynecol 2009;114(6):1341-5. doi: 10.1097/AOG.0b013e3181c3020d [published Online First: 2009/11/26]

Contributors

Tina Poklepović Peričić

Medical School

Sarah Tanveer

Jump to navigation

Cochrane Training

Chapter 1: starting a review.

Toby J Lasserson, James Thomas, Julian PT Higgins

Key Points:

Systematic reviews address a need for health decision makers to be able to access high quality, relevant, accessible and up-to-date information.
Systematic reviews aim to minimize bias through the use of pre-specified research questions and methods that are documented in protocols, and by basing their findings on reliable research.
Systematic reviews should be conducted by a team that includes domain expertise and methodological expertise, who are free of potential conflicts of interest.
People who might make – or be affected by – decisions around the use of interventions should be involved in important decisions about the review.
Good data management, project management and quality assurance mechanisms are essential for the completion of a successful systematic review.

Cite this chapter as: Lasserson TJ, Thomas J, Higgins JPT. Chapter 1: Starting a review. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

1.1 Why do a systematic review?

Systematic reviews were developed out of a need to ensure that decisions affecting people’s lives can be informed by an up-to-date and complete understanding of the relevant research evidence. With the volume of research literature growing at an ever-increasing rate, it is impossible for individual decision makers to assess this vast quantity of primary research to enable them to make the most appropriate healthcare decisions that do more good than harm. By systematically assessing this primary research, systematic reviews aim to provide an up-to-date summary of the state of research knowledge on an intervention, diagnostic test, prognostic factor or other health or healthcare topic. Systematic reviews address the main problem with ad hoc searching and selection of research, namely that of bias. Just as primary research studies use methods to avoid bias, so should summaries and syntheses of that research.

A systematic review attempts to collate all the empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question. It uses explicit, systematic methods that are selected with a view to minimizing bias, thus providing more reliable findings from which conclusions can be drawn and decisions made (Antman et al 1992, Oxman and Guyatt 1993). Systematic review methodology, pioneered and developed by Cochrane, sets out a highly structured, transparent and reproducible methodology (Chandler and Hopewell 2013). This involves: the a priori specification of a research question; clarity on the scope of the review and which studies are eligible for inclusion; making every effort to find all relevant research and to ensure that issues of bias in included studies are accounted for; and analysing the included studies in order to draw conclusions based on all the identified research in an impartial and objective way.

This Handbook is about systematic reviews on the effects of interventions, and specifically about methods used by Cochrane to undertake them. Cochrane Reviews use primary research to generate new knowledge about the effects of an intervention (or interventions) used in clinical, public health or policy settings. They aim to provide users with a balanced summary of the potential benefits and harms of interventions and give an indication of how certain they can be of the findings. They can also compare the effectiveness of different interventions with one another and so help users to choose the most appropriate intervention in particular situations. The primary purpose of Cochrane Reviews is therefore to inform people making decisions about health or health care.

Systematic reviews are important for other reasons. New research should be designed or commissioned only if it does not unnecessarily duplicate existing research (Chalmers et al 2014). Therefore, a systematic review should typically be undertaken before embarking on new primary research. Such a review will identify current and ongoing studies, as well as indicate where specific gaps in knowledge exist, or evidence is lacking; for example, where existing studies have not used outcomes that are important to users of research (Macleod et al 2014). A systematic review may also reveal limitations in the conduct of previous studies that might be addressed in the new study or studies.

Systematic reviews are important, often rewarding and, at times, exciting research projects. They offer the opportunity for authors to make authoritative statements about the extent of human knowledge in important areas and to identify priorities for further research. They sometimes cover issues high on the political agenda and receive attention from the media. Conducting research with these impacts is not without its challenges, however, and completing a high-quality systematic review is often demanding and time-consuming. In this chapter we introduce some of the key considerations for potential review authors who are about to start a systematic review.

1.2 What is the review question?

Getting the research question right is critical for the success of a systematic review. Review authors should ensure that the review addresses an important question to those who are expected to use and act upon its conclusions.

We discuss the formulation of questions in detail in Chapter 2 . For a question about the effects of an intervention, the PICO approach is usually used, which is an acronym for Population, Intervention, Comparison(s) and Outcome. Reviews may have additional questions, for example about how interventions were implemented, economic issues, equity issues or patient experience.

To ensure that the review addresses a relevant question in a way that benefits users, it is important to ensure wide input. In most cases, question formulation should therefore be informed by people with various relevant – but potentially different – perspectives (see Chapter 2, Section 2.4 ).

1.3 Who should do a systematic review?

Systematic reviews should be undertaken by a team. Indeed, Cochrane will not publish a review that is proposed to be undertaken by a single person. Working as a team not only spreads the effort, but ensures that tasks such as the selection of studies for eligibility, data extraction and rating the certainty of the evidence will be performed by at least two people independently, minimizing the likelihood of errors. First-time review authors are encouraged to work with others who are experienced in the process of systematic reviews and to attend relevant training.

Review teams must include expertise in the topic area under review. Topic expertise should not be overly narrow, to ensure that all relevant perspectives are considered. Perspectives from different disciplines can help to avoid assumptions or terminology stemming from an over-reliance on a single discipline. Review teams should also include expertise in systematic review methodology, including statistical expertise.

Arguments have been made that methodological expertise is sufficient to perform a review, and that content expertise should be avoided because of the risk of preconceptions about the effects of interventions (Gøtzsche and Ioannidis 2012). However, it is important that both topic and methodological expertise is present to ensure a good mix of skills, knowledge and objectivity, because topic expertise provides important insight into the implementation of the intervention(s), the nature of the condition being treated or prevented, the relationships between outcomes measured, and other factors that may have an impact on decision making.

A Cochrane Review should represent an independent assessment of the evidence and avoiding financial and non-financial conflicts of interest often requires careful management. It will be important to consider if there are any relevant interests that may constitute a conflict of interest. There are situations where employment, holding of patents and other financial support should prevent people joining an author team. Funding of Cochrane Reviews by commercial organizations with an interest in the outcome of the review is not permitted. To ensure that any issues are identified early in the process, authors planning Cochrane Reviews should consult the Conflict of Interest Policy . Authors should make complete declarations of interest before registration of the review, and refresh these annually thereafter until publication and just prior to publication of the protocol and the review. For authors of review updates, this must be done at the time of the decision to update the review, annually thereafter until publication, and just prior to publication. Authors should also update declarations of interest at any point when their circumstances change.

1.3.1 Involving consumers and other stakeholders

Because the priorities of decision makers and consumers may be different from those of researchers, it is important that review authors consider carefully what questions are important to these different stakeholders. Systematic reviews are more likely to be relevant to a broad range of end users if they are informed by the involvement of people with a range of experiences, in terms of both the topic and the methodology (Thomas et al 2004, Rees and Oliver 2017). Engaging consumers and other stakeholders, such as policy makers, research funders and healthcare professionals, increases relevance, promotes mutual learning, improved uptake and decreases research waste.

Mapping out all potential stakeholders specific to the review question is a helpful first step to considering who might be invited to be involved in a review. Stakeholders typically include: patients and consumers; consumer advocates; policy makers and other public officials; guideline developers; professional organizations; researchers; funders of health services and research; healthcare practitioners, and, on occasion, journalists and other media professionals. Balancing seniority, credibility within the given field, and diversity should be considered. Review authors should also take account of the needs of resource-poor countries and regions in the review process (see Chapter 16 ) and invite appropriate input on the scope of the review and the questions it will address.

It is established good practice to ensure that consumers are involved and engaged in health research, including systematic reviews. Cochrane uses the term ‘consumers’ to refer to a wide range of people, including patients or people with personal experience of a healthcare condition, carers and family members, representatives of patients and carers, service users and members of the public. In 2017, a Statement of Principles for consumer involvement in Cochrane was agreed. This seeks to change the culture of research practice to one where both consumers and other stakeholders are joint partners in research from planning, conduct, and reporting to dissemination. Systematic reviews that have had consumer involvement should be more directly applicable to decision makers than those that have not (see online Chapter II ).

1.3.2 Working with consumers and other stakeholders

Methods for working with consumers and other stakeholders include surveys, workshops, focus groups and involvement in advisory groups. Decisions about what methods to use will typically be based on resource availability, but review teams should be aware of the merits and limitations of such methods. Authors will need to decide who to involve and how to provide adequate support for their involvement. This can include financial reimbursement, the provision of training, and stating clearly expectations of involvement, possibly in the form of terms of reference.

While a small number of consumers or other stakeholders may be part of the review team and become co-authors of the subsequent review, it is sometimes important to bring in a wider range of perspectives and to recognize that not everyone has the capacity or interest in becoming an author. Advisory groups offer a convenient approach to involving consumers and other relevant stakeholders, especially for topics in which opinions differ. Important points to ensure successful involvement include the following.

The review team should co-ordinate the input of the advisory group to inform key review decisions.
The advisory group’s input should continue throughout the systematic review process to ensure relevance of the review to end users is maintained.
Advisory group membership should reflect the breadth of the review question, and consideration should be given to involving vulnerable and marginalized people (Steel 2004) to ensure that conclusions on the value of the interventions are well-informed and applicable to all groups in society (see Chapter 16 ).

Templates such as terms of reference, job descriptions, or person specifications for an advisory group help to ensure clarity about the task(s) required and are available from INVOLVE . The website also gives further information on setting and organizing advisory groups. See also the Cochrane training website for further resources to support consumer involvement.

1.4 The importance of reliability

Systematic reviews aim to be an accurate representation of the current state of knowledge about a given issue. As understanding improves, the review can be updated. Nevertheless, it is important that the review itself is accurate at the time of publication. There are two main reasons for this imperative for accuracy. First, health decisions that affect people’s lives are increasingly taken based on systematic review findings. Current knowledge may be imperfect, but decisions will be better informed when taken in the light of the best of current knowledge. Second, systematic reviews form a critical component of legal and regulatory frameworks; for example, drug licensing or insurance coverage. Here, systematic reviews also need to hold up as auditable processes for legal examination. As systematic reviews need to be both correct, and be seen to be correct, detailed evidence-based methods have been developed to guide review authors as to the most appropriate procedures to follow, and what information to include in their reports to aid auditability.

1.4.1 Expectations for the conduct and reporting of Cochrane Reviews

Cochrane has developed methodological expectations for the conduct, reporting and updating of systematic reviews of interventions (MECIR) and their plain language summaries ( Plain Language Expectations for Authors of Cochrane Summaries ; PLEACS). Developed collaboratively by methodologists and Cochrane editors, they are intended to describe the desirable attributes of a Cochrane Review. The expectations are not all relevant at the same stage of review conduct, so care should be taken to identify those that are relevant at specific points during the review. Different methods should be used at different stages of the review in terms of the planning, conduct, reporting and updating of the review.

Each expectation has a title, a rationale and an elaboration. For the purposes of publication of a review with Cochrane, each has the status of either ‘mandatory’ or ‘highly desirable’. Items described as mandatory are expected to be applied, and if they are not then an appropriate justification should be provided; failure to implement such items may be used as a basis for deciding not to publish a review in the Cochrane Database of Systematic Reviews (CDSR). Items described as highly desirable should generally be implemented, but there are reasonable exceptions and justifications are not required.

All MECIR expectations for the conduct of a review are presented in the relevant chapters of this Handbook . Expectations for reporting of completed reviews (including PLEACS) are described in online Chapter III . The recommendations provided in the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) Statement have been incorporated into the Cochrane reporting expectations, ensuring compliance with the PRISMA recommendations and summarizing attributes of reporting that should allow a full assessment of the methods and findings of the review (Moher et al 2009).

1.5 Protocol development

Preparing a systematic review is complex and involves many judgements. To minimize the potential for bias in the review process, these judgements should be made as far as possible in ways that do not depend on the findings of the studies included in the review. Review authors’ prior knowledge of the evidence may, for example, influence the definition of a systematic review question, the choice of criteria for study eligibility, or the pre-specification of intervention comparisons and outcomes to analyse. It is important that the methods to be used should be established and documented in advance (see MECIR Box 1.5.a , MECIR Box 1.5.b and MECIR Box 1.5.c ).

Publication of a protocol for a review that is written without knowledge of the available studies reduces the impact of review authors’ biases, promotes transparency of methods and processes, reduces the potential for duplication, allows peer review of the planned methods before they have been completed, and offers an opportunity for the review team to plan resources and logistics for undertaking the review itself. All chapters in the Handbook should be consulted when drafting the protocol. Since systematic reviews are by their nature retrospective, an element of knowledge of the evidence is often inevitable. This is one reason why non-content experts such as methodologists should be part of the review team (see Section 1.3 ). Two exceptions to the retrospective nature of a systematic review are a meta-analysis of a prospectively planned series of trials and some living systematic reviews, as described in Chapter 22 .

The review question should determine the methods used in the review, and not vice versa. The question may concern a relatively straightforward comparison of one treatment with another; or it may necessitate plans to compare different treatments as part of a network meta-analysis, or assess differential effects of an intervention in different populations or delivered in different ways.

The protocol sets out the context in which the review is being conducted. It presents an opportunity to develop ideas that are foundational for the review. This concerns, most explicitly, definition of the eligibility criteria such as the study participants and the choice of comparators and outcomes. The eligibility criteria may also be defined following the development of a logic model (or an articulation of the aspects of an extent logic model that the review is addressing) to explain how the intervention might work (see Chapter 2, Section 2.5.1 ).

MECIR Box 1.5.a Relevant expectations for conduct of intervention reviews

: Planning the search ( )

Searches should be motivated directly by the eligibility criteria for the review, and it is important that all types of eligible studies are considered when planning the search. If searches are restricted by publication status or by language of publication, there is a possibility of publication bias, or language bias (whereby the language of publication is selected in a way that depends on the findings of the study), or both. Removing language restrictions in English language databases is not a good substitute for searching non-English language journals and databases.

A key purpose of the protocol is to make plans to minimize bias in the eventual findings of the review. Reliable synthesis of available evidence requires a planned, systematic approach. Threats to the validity of systematic reviews can come from the studies they include or the process by which reviews are conducted. Biases within the studies can arise from the method by which participants are allocated to the intervention groups, awareness of intervention group assignment, and the collection, analysis and reporting of data. Methods for examining these issues should be specified in the protocol. Review processes can generate bias through a failure to identify an unbiased (and preferably complete) set of studies, and poor quality assurance throughout the review. The availability of research may be influenced by the nature of the results (i.e. reporting bias). To reduce the impact of this form of bias, searching may need to include unpublished sources of evidence (Dwan et al 2013) ( MECIR Box 1.5.b ).

MECIR Box 1.5.b Relevant expectations for the conduct of intervention reviews

Planning the assessment of risk of bias in included studies ( )

Predefining the methods and criteria for assessing risk of bias is important since analysis or interpretation of the review findings may be affected by the judgements made during this process. For randomized trials, use of the Cochrane risk-of-bias tool is Mandatory, so it is sufficient (and easiest) simply to refer to the definitions of low risk, unclear risk and high risk of bias provided in the .

Developing a protocol for a systematic review has benefits beyond reducing bias. Investing effort in designing a systematic review will make the process more manageable and help to inform key priorities for the review. Defining the question, referring to it throughout, and using appropriate methods to address the question focuses the analysis and reporting, ensuring the review is most likely to inform treatment decisions for funders, policy makers, healthcare professionals and consumers. Details of the planned analyses, including investigations of variability across studies, should be specified in the protocol, along with methods for interpreting the results through the systematic consideration of factors that affect confidence in estimates of intervention effect ( MECIR Box 1.5.c ).

MECIR Box 1.5.c Relevant expectations for conduct of intervention reviews

Planning the synthesis of results ( )
.	Predefining the synthesis methods, particularly the statistical methods, is important, since analysis or interpretation of the review findings may be affected by the judgements made during this process.
Planning sub-group analyses ( )
.	Pre-specification reduces the risk that large numbers of undirected subgroup analyses will lead to spurious explanations of heterogeneity.
Planning the GRADE assessment and ‘Summary of findings’ table ( )
.	Methods for assessing the certainty of evidence for the most important outcomes in the review need to be pre-specified. In ‘Summary of findings’ tables the most important feature is to predefine the choice of outcomes in order to guard against selective presentation of results in the review. The table should include the essential outcomes for decision making (typically up to seven), which generally should not include surrogate or interim outcomes. The choice of outcomes should not be based on any anticipated or observed magnitude of effect, or because they are likely to have been addressed in the studies to be reviewed.

While the intention should be that a review will adhere to the published protocol, changes in a review protocol are sometimes necessary. This is also the case for a protocol for a randomized trial, which must sometimes be changed to adapt to unanticipated circumstances such as problems with participant recruitment, data collection or event rates. While every effort should be made to adhere to a predetermined protocol, this is not always possible or appropriate. It is important, however, that changes in the protocol should not be made based on how they affect the outcome of the research study, whether it is a randomized trial or a systematic review. Post hoc decisions made when the impact on the results of the research is known, such as excluding selected studies from a systematic review, or changing the statistical analysis, are highly susceptible to bias and should therefore be avoided unless there are reasonable grounds for doing this.

Enabling access to a protocol through publication (all Cochrane Protocols are published in the CDSR ) and registration on the PROSPERO register of systematic reviews reduces duplication of effort, research waste, and promotes accountability. Changes to the methods outlined in the protocol should be transparently declared.

This Handbook provides details of the systematic review methods developed or selected by Cochrane. They are intended to address the need for rigour, comprehensiveness and transparency in preparing a Cochrane systematic review. All relevant chapters – including those describing procedures to be followed in the later stages of the review – should be consulted during the preparation of the protocol. A more specific description of the structure of Cochrane Protocols is provide in online Chapter II .

1.6 Data management and quality assurance

Systematic reviews should be replicable, and retaining a record of the inclusion decisions, data collection, transformations or adjustment of data will help to establish a secure and retrievable audit trail. They can be operationally complex projects, often involving large research teams operating in different sites across the world. Good data management processes are essential to ensure that data are not inadvertently lost, facilitating the identification and correction of errors and supporting future efforts to update and maintain the review. Transparent reporting of review decisions enables readers to assess the reliability of the review for themselves.

Review management software, such as Covidence and EPPI-Reviewer , can be used to assist data management and maintain consistent and standardized records of decisions made throughout the review. These tools offer a central repository for review data that can be accessed remotely throughout the world by members of the review team. They record independent assessment of studies for inclusion, risk of bias and extraction of data, enabling checks to be made later in the process if needed. Research has shown that even experienced reviewers make mistakes and disagree with one another on risk-of-bias assessments, so it is particularly important to maintain quality assurance here, despite its cost in terms of author time. As more sophisticated information technology tools begin to be deployed in reviews (see Chapter 4, Section 4.6.6.2 and Chapter 22, Section 22.2.4 ), it is increasingly apparent that all review data – including the initial decisions about study eligibility – have value beyond the scope of the individual review. For example, review updates can be made more efficient through (semi-) automation when data from the original review are available for machine learning.

1.7 Chapter information

Authors: Toby J Lasserson, James Thomas, Julian PT Higgins

Acknowledgements: This chapter builds on earlier versions of the Handbook . We would like to thank Ruth Foxlee, Richard Morley, Soumyadeep Bhaumik, Mona Nasser, Dan Fox and Sally Crowe for their contributions to Section 1.3 .

Funding: JT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. JPTH is a member of the NIHR Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. JPTH received funding from National Institute for Health Research Senior Investigator award NF-SI-0617-10145. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

1.8 References

Antman E, Lau J, Kupelnick B, Mosteller F, Chalmers T. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatment for myocardial infarction. JAMA 1992; 268 : 240–248.

Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet 2014; 383 : 156–165.

Chandler J, Hopewell S. Cochrane methods – twenty years experience in developing systematic review methods. Systematic Reviews 2013; 2 : 76.

Dwan K, Gamble C, Williamson PR, Kirkham JJ, Reporting Bias Group. Systematic review of the empirical evidence of study publication bias and outcome reporting bias: an updated review. PloS One 2013; 8 : e66844.

Gøtzsche PC, Ioannidis JPA. Content area experts as authors: helpful or harmful for systematic reviews and meta-analyses? BMJ 2012; 345 .

Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet 2014; 383 : 101–104.

Moher D, Liberati A, Tetzlaff J, Altman D, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Medicine 2009; 6 : e1000097.

Oxman A, Guyatt G. The science of reviewing research. Annals of the New York Academy of Sciences 1993; 703 : 125–133.

Rees R, Oliver S. Stakeholder perspectives and participation in reviews. In: Gough D, Oliver S, Thomas J, editors. An Introduction to Systematic Reviews . 2nd ed. London: Sage; 2017. p. 17–34.

Steel R. Involving marginalised and vulnerable people in research: a consultation document (2nd revision). INVOLVE; 2004.

Thomas J, Harden A, Oakley A, Oliver S, Sutcliffe K, Rees R, Brunton G, Kavanagh J. Integrating qualitative research with trials in systematic reviews. BMJ 2004; 328 : 1010–1012.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

Knowledge Base

Methodology

Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

Formulate a research question
Develop a protocol
Search for all relevant studies
Apply the selection criteria
Extract the data
Synthesize the data
Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism. Run a free check.

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
Access to databases and journal archives. Often, your educational institution provides you with access.
Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

They minimize research bias by considering all available evidence and evaluating each study for bias.
Their methods are transparent , so they can be scrutinized by others.
They’re thorough : they summarize all available evidence.
They can be replicated and updated by others.

Systematic reviews also have a few cons .

They’re time-consuming .
They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

Allow you to more effectively communicate your research to other researchers and practitioners
Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

Population(s) or problem(s)
Intervention(s)
Comparison(s)

You can rearrange these four components to write your research question:

What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

Type of study design(s)
The population of patients with eczema
The intervention of probiotics
In comparison to no treatment, placebo , or non-probiotic treatment
The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
Randomized control trials, a type of study design

Their research question was:

What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

Background information : Provide the context of the research question, including why it’s important.
Research objective (s) : Rephrase your research question as an objective.
Selection criteria: State how you’ll decide which studies to include or exclude from your review.
Search strategy: Discuss your plan for finding studies.
Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
Handsearch: Conference proceedings and reference lists of articles
Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

Abstract : A summary of the review
Introduction : Including the rationale and objectives
Methods : Including the selection criteria, search method, data extraction method, and synthesis method
Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
Discussion : Including interpretation of the results and limitations of the review
Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

Student’s t -distribution
Normal distribution
Null and Alternative Hypotheses
Chi square tests
Confidence interval
Quartiles & Quantiles
Cluster sampling
Stratified sampling
Data cleansing
Reproducibility vs Replicability
Peer review
Prospective cohort study

Research bias

Implicit bias
Cognitive bias
Placebo effect
Hawthorne effect
Hindsight bias
Affect heuristic
Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other academic texts , with an introduction , a main body, and a conclusion .

An annotated bibliography is a list of source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a paper .

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved August 21, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

Easy guide to conducting a systematic review

Affiliations.

1 Discipline of Child and Adolescent Health, University of Sydney, Sydney, New South Wales, Australia.
2 Department of Nephrology, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
3 Education Department, The Children's Hospital at Westmead, Sydney, New South Wales, Australia.
PMID: 32364273
DOI: 10.1111/jpc.14853

A systematic review is a type of study that synthesises research that has been conducted on a particular topic. Systematic reviews are considered to provide the highest level of evidence on the hierarchy of evidence pyramid. Systematic reviews are conducted following rigorous research methodology. To minimise bias, systematic reviews utilise a predefined search strategy to identify and appraise all available published literature on a specific topic. The meticulous nature of the systematic review research methodology differentiates a systematic review from a narrative review (literature review or authoritative review). This paper provides a brief step by step summary of how to conduct a systematic review, which may be of interest for clinicians and researchers.

Keywords: research; research design; systematic review.

PubMed Disclaimer

Publication types

Search in MeSH

Related information

Linkout - more resources, full text sources.

Ovid Technologies, Inc.

Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

A-Z Publications

Annual Review of Psychology

Volume 70, 2019, review article, how to do a systematic review: a best practice guide for conducting and reporting narrative reviews, meta-analyses, and meta-syntheses.

Andy P. Siddaway 1 , Alex M. Wood 2 , and Larry V. Hedges 3
View Affiliations Hide Affiliations Affiliations: 1 Behavioural Science Centre, Stirling Management School, University of Stirling, Stirling FK9 4LA, United Kingdom; email: [email protected] 2 Department of Psychological and Behavioural Science, London School of Economics and Political Science, London WC2A 2AE, United Kingdom 3 Department of Statistics, Northwestern University, Evanston, Illinois 60208, USA; email: [email protected]
Vol. 70:747-770 (Volume publication date January 2019) https://doi.org/10.1146/annurev-psych-010418-102803
First published as a Review in Advance on August 08, 2018
Copyright © 2019 by Annual Reviews. All rights reserved

Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to draw broad theoretical conclusions about what a literature means, linking theory to evidence and evidence to theory. This guide describes how to plan, conduct, organize, and present a systematic review of quantitative (meta-analysis) or qualitative (narrative review, meta-synthesis) information. We outline core standards and principles and describe commonly encountered problems. Although this guide targets psychological scientists, its high level of abstraction makes it potentially relevant to any subject area or discipline. We argue that systematic reviews are a key methodology for clarifying whether and how research findings replicate and for explaining possible inconsistencies, and we call for researchers to conduct systematic reviews to help elucidate whether there is a replication crisis.

Article metrics loading...

Full text loading...

Literature Cited

APA Publ. Commun. Board Work. Group J. Artic. Rep. Stand. 2008 . Reporting standards for research in psychology: Why do we need them? What might they be?. Am. Psychol . 63 : 848– 49 [Google Scholar]
Baumeister RF 2013 . Writing a literature review. The Portable Mentor: Expert Guide to a Successful Career in Psychology MJ Prinstein, MD Patterson 119– 32 New York: Springer, 2nd ed.. [Google Scholar]
Baumeister RF , Leary MR 1995 . The need to belong: desire for interpersonal attachments as a fundamental human motivation. Psychol. Bull. 117 : 497– 529 [Google Scholar]
Baumeister RF , Leary MR 1997 . Writing narrative literature reviews. Rev. Gen. Psychol. 3 : 311– 20 Presents a thorough and thoughtful guide to conducting narrative reviews. [Google Scholar]
Bem DJ 1995 . Writing a review article for Psychological Bulletin. Psychol . Bull 118 : 172– 77 [Google Scholar]
Borenstein M , Hedges LV , Higgins JPT , Rothstein HR 2009 . Introduction to Meta-Analysis New York: Wiley Presents a comprehensive introduction to meta-analysis. [Google Scholar]
Borenstein M , Higgins JPT , Hedges LV , Rothstein HR 2017 . Basics of meta-analysis: I 2 is not an absolute measure of heterogeneity. Res. Synth. Methods 8 : 5– 18 [Google Scholar]
Braver SL , Thoemmes FJ , Rosenthal R 2014 . Continuously cumulating meta-analysis and replicability. Perspect. Psychol. Sci. 9 : 333– 42 [Google Scholar]
Bushman BJ 1994 . Vote-counting procedures. The Handbook of Research Synthesis H Cooper, LV Hedges 193– 214 New York: Russell Sage Found. [Google Scholar]
Cesario J 2014 . Priming, replication, and the hardest science. Perspect. Psychol. Sci. 9 : 40– 48 [Google Scholar]
Chalmers I 2007 . The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the importance of systematic reviews. Treating Individuals: From Randomised Trials to Personalised Medicine PM Rothwell 37– 58 London: Lancet [Google Scholar]
Cochrane Collab. 2003 . Glossary Rep., Cochrane Collab. London: http://community.cochrane.org/glossary Presents a comprehensive glossary of terms relevant to systematic reviews. [Google Scholar]
Cohn LD , Becker BJ 2003 . How meta-analysis increases statistical power. Psychol. Methods 8 : 243– 53 [Google Scholar]
Cooper HM 2003 . Editorial. Psychol. Bull. 129 : 3– 9 [Google Scholar]
Cooper HM 2016 . Research Synthesis and Meta-Analysis: A Step-by-Step Approach Thousand Oaks, CA: Sage, 5th ed.. Presents a comprehensive introduction to research synthesis and meta-analysis. [Google Scholar]
Cooper HM , Hedges LV , Valentine JC 2009 . The Handbook of Research Synthesis and Meta-Analysis New York: Russell Sage Found, 2nd ed.. [Google Scholar]
Cumming G 2014 . The new statistics: why and how. Psychol. Sci. 25 : 7– 29 Discusses the limitations of null hypothesis significance testing and viable alternative approaches. [Google Scholar]
Earp BD , Trafimow D 2015 . Replication, falsification, and the crisis of confidence in social psychology. Front. Psychol. 6 : 621 [Google Scholar]
Etz A , Vandekerckhove J 2016 . A Bayesian perspective on the reproducibility project: psychology. PLOS ONE 11 : e0149794 [Google Scholar]
Ferguson CJ , Brannick MT 2012 . Publication bias in psychological science: prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychol. Methods 17 : 120– 28 [Google Scholar]
Fleiss JL , Berlin JA 2009 . Effect sizes for dichotomous data. The Handbook of Research Synthesis and Meta-Analysis H Cooper, LV Hedges, JC Valentine 237– 53 New York: Russell Sage Found, 2nd ed.. [Google Scholar]
Garside R 2014 . Should we appraise the quality of qualitative research reports for systematic reviews, and if so, how. Innovation 27 : 67– 79 [Google Scholar]
Hedges LV , Olkin I 1980 . Vote count methods in research synthesis. Psychol. Bull. 88 : 359– 69 [Google Scholar]
Hedges LV , Pigott TD 2001 . The power of statistical tests in meta-analysis. Psychol. Methods 6 : 203– 17 [Google Scholar]
Higgins JPT , Green S 2011 . Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 London: Cochrane Collab. Presents comprehensive and regularly updated guidelines on systematic reviews. [Google Scholar]
John LK , Loewenstein G , Prelec D 2012 . Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 : 524– 32 [Google Scholar]
Juni P , Witschi A , Bloch R , Egger M 1999 . The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 282 : 1054– 60 [Google Scholar]
Klein O , Doyen S , Leys C , Magalhães de Saldanha da Gama PA , Miller S et al. 2012 . Low hopes, high expectations: expectancy effects and the replicability of behavioral experiments. Perspect. Psychol. Sci. 7 : 6 572– 84 [Google Scholar]
Lau J , Antman EM , Jimenez-Silva J , Kupelnick B , Mosteller F , Chalmers TC 1992 . Cumulative meta-analysis of therapeutic trials for myocardial infarction. N. Engl. J. Med. 327 : 248– 54 [Google Scholar]
Light RJ , Smith PV 1971 . Accumulating evidence: procedures for resolving contradictions among different research studies. Harvard Educ. Rev. 41 : 429– 71 [Google Scholar]
Lipsey MW , Wilson D 2001 . Practical Meta-Analysis London: Sage Comprehensive and clear explanation of meta-analysis. [Google Scholar]
Matt GE , Cook TD 1994 . Threats to the validity of research synthesis. The Handbook of Research Synthesis H Cooper, LV Hedges 503– 20 New York: Russell Sage Found. [Google Scholar]
Maxwell SE , Lau MY , Howard GS 2015 . Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. Am. Psychol. 70 : 487– 98 [Google Scholar]
Moher D , Hopewell S , Schulz KF , Montori V , Gøtzsche PC et al. 2010 . CONSORT explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340 : c869 [Google Scholar]
Moher D , Liberati A , Tetzlaff J , Altman DG PRISMA Group. 2009 . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339 : 332– 36 Comprehensive reporting guidelines for systematic reviews. [Google Scholar]
Morrison A , Polisena J , Husereau D , Moulton K , Clark M et al. 2012 . The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. Int. J. Technol. Assess. Health Care 28 : 138– 44 [Google Scholar]
Nelson LD , Simmons J , Simonsohn U 2018 . Psychology's renaissance. Annu. Rev. Psychol. 69 : 511– 34 [Google Scholar]
Noblit GW , Hare RD 1988 . Meta-Ethnography: Synthesizing Qualitative Studies Newbury Park, CA: Sage [Google Scholar]
Olivo SA , Macedo LG , Gadotti IC , Fuentes J , Stanton T , Magee DJ 2008 . Scales to assess the quality of randomized controlled trials: a systematic review. Phys. Ther. 88 : 156– 75 [Google Scholar]
Open Sci. Collab. 2015 . Estimating the reproducibility of psychological science. Science 349 : 943 [Google Scholar]
Paterson BL , Thorne SE , Canam C , Jillings C 2001 . Meta-Study of Qualitative Health Research: A Practical Guide to Meta-Analysis and Meta-Synthesis Thousand Oaks, CA: Sage [Google Scholar]
Patil P , Peng RD , Leek JT 2016 . What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspect. Psychol. Sci. 11 : 539– 44 [Google Scholar]
Rosenthal R 1979 . The “file drawer problem” and tolerance for null results. Psychol. Bull. 86 : 638– 41 [Google Scholar]
Rosnow RL , Rosenthal R 1989 . Statistical procedures and the justification of knowledge in psychological science. Am. Psychol. 44 : 1276– 84 [Google Scholar]
Sanderson S , Tatt ID , Higgins JP 2007 . Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int. J. Epidemiol. 36 : 666– 76 [Google Scholar]
Schreiber R , Crooks D , Stern PN 1997 . Qualitative meta-analysis. Completing a Qualitative Project: Details and Dialogue JM Morse 311– 26 Thousand Oaks, CA: Sage [Google Scholar]
Shrout PE , Rodgers JL 2018 . Psychology, science, and knowledge construction: broadening perspectives from the replication crisis. Annu. Rev. Psychol. 69 : 487– 510 [Google Scholar]
Stroebe W , Strack F 2014 . The alleged crisis and the illusion of exact replication. Perspect. Psychol. Sci. 9 : 59– 71 [Google Scholar]
Stroup DF , Berlin JA , Morton SC , Olkin I , Williamson GD et al. 2000 . Meta-analysis of observational studies in epidemiology (MOOSE): a proposal for reporting. JAMA 283 : 2008– 12 [Google Scholar]
Thorne S , Jensen L , Kearney MH , Noblit G , Sandelowski M 2004 . Qualitative meta-synthesis: reflections on methodological orientation and ideological agenda. Qual. Health Res. 14 : 1342– 65 [Google Scholar]
Tong A , Flemming K , McInnes E , Oliver S , Craig J 2012 . Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med. Res. Methodol. 12 : 181– 88 [Google Scholar]
Trickey D , Siddaway AP , Meiser-Stedman R , Serpell L , Field AP 2012 . A meta-analysis of risk factors for post-traumatic stress disorder in children and adolescents. Clin. Psychol. Rev. 32 : 122– 38 [Google Scholar]
Valentine JC , Biglan A , Boruch RF , Castro FG , Collins LM et al. 2011 . Replication in prevention science. Prev. Sci. 12 : 103– 17 [Google Scholar]
Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, job burnout, executive functions, social cognitive theory: an agentic perspective, on happiness and human potentials: a review of research on hedonic and eudaimonic well-being, sources of method bias in social science research and recommendations on how to control it, mediation analysis, missing data analysis: making it work in the real world, grounded cognition, personality structure: emergence of the five-factor model, motivational beliefs, values, and goals.

The Graduate Health & Life Sciences Research Library at Georgetown University Medical Center

Systematic reviews.

Should I do a systematic review?
Writing the Protocol
Building a Systematic Search
Where to Search
Managing Project Data
How can a DML librarian help?

Guides and Standards

The Cochrane Handbook The Cochrane Handbook has become the de facto standard for planning and carrying out a systematic review. Chapter 6, Searching for Studies, is most helpful in planning your review.
Finding What Works in Health Care: Standards for Systematic Reviews The IOM standards promote objective, transparent, and scientifically valid systematic reviews. They address the entire systematic review process, from locating, screening, and selecting studies for the review, to synthesizing the findings (including meta-analysis) and assessing the overall quality of the body of evidence, to producing the final review report.
PRISMA Standards The Preferred Reporting Items for Systematic Reviews and Meta-Analyses is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. A 27-item checklist, PRISMA focuses on randomized trials but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.

What is a systematic review?

A systematic literature review is a research methodology designed to answer a focused research question. Authors conduct a methodical and comprehensive literature synthesis focused on a well-formulated research question. Its aim is to identify and synthesize all of the scholarly research on a particular topic, including both published and unpublished studies. Systematic reviews are conducted in an unbiased, reproducible way to provide evidence for practice and policy-making and identify gaps in research. Every step of the review, including the search, must be documented for reproducibility.

Researchers in medicine may be most familiar with Cochrane Reviews, which synthesize randomized controlled trials to evaluate specific medical interventions. Systematic reviews are conducted in many other fields, though the type of evidence analyzed varies with the research question.

When to use systematic review methodology

Systematic reviews require more time and manpower than traditional literature reviews. Before beginning a systematic review, researchers should address these questions:

Is there is enough literature published on the topic to warrant a review?

Systematic reviews are designed to distill the evidence from many studies into actionable insights. Is there a body of evidence available to analyze, or does more primary research need to be done?

Can your research question be answered by a systematic review?

Systematic review questions should be specific and clearly defined. Questions that fit the PICO (problem/patient, intervention, comparison, outcome) format are usually well-suited for the systematic review methodology. The research question determines the search strategy, inclusion criteria, and data that you extract from the selected studies, so it should be clearly defined at the start of the review process.

Do you have a protocol outlining the review plan?

The protocol is the roadmap for the review project. A good protocol outlines study methodology, includes the rationale for the systematic review, and describes the key question broken into PICO components. It is also a good place to plan out inclusion/exclusion criteria, databases that will be searched, data abstraction and management methods, and how the studies will be assessed for methodological quality.

Do you have a team of experts?

A systematic review is team effort. Having multiple reviewers minimizes bias and strengthens analysis. Teams are often composed of subject experts, two or more literature screeners, a librarian to conduct the search, and a statistician to analyze the data.

Do you have the time that it takes to properly conduct a systematic review?

Systematic reviews typically take 12-18 months.

Do you have a method for discerning bias?

There are many types of bias, including selection, performance, & reporting bias, and assessing the risk of bias of individual studies is an important part of your study design.

Can you afford to have articles in languages other than English translated?

You should include all relevant studies in your systematic review, regardless of the language they were published in, so as to avoid language bias.

Which review is right for you?

If your project does not meet the above criteria, there are many more options for conducting a synthesis of the literature. The chart below highlights several review methodologies. Reproduced from: Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Info Libr J. 2009 Jun;26(2):91-108. doi: 10.1111/j.1471-1842.2009.00848.x . Review. PubMed PMID: 19490148

Label	Description	Search	Appraisal	Synthesis	Analysis
Critical review	Aims to demonstrate writer has extensively researched literature and critically evaluated its quality. Goes beyond mere description to include degree of analysis and conceptual innovation. Typically results in hypothesis or model.	Seeks to identify significant items in the field.	No formal quality assessment. Attempts to evaluate according to contribution.	Typically narrative, perhaps conceptual or chronological.	Significant component: seeks to identify conceptual contribution to embody existing or derive new theory.
Literature review	Generic term: a search for published materials that provide examination of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness. May include research findings.	May or may not include comprehensive searching.	May or may not include quality assessment.	Typically narrative.	Analysis may be chronological, conceptual, thematic, etc.
Mapping review/systematic map	Maps out and categorizes existing literature from which to commission further reviews and/or primary research by identifying gaps in research literature.	Completeness of searching determined by time/scope constraints.	No formal quality assessment.	May be graphical and tabular.	Characterizes quantity and quality of literature, perhaps by study design and other key features. May identify need for primary or secondary research.
Meta-analysis	Technique that statistically combines the results of quantitative studies to provide a more precise effect of the results.	Aims for exhaustive searching. May use funnel plot to assess completeness.	Quality assessment may determine inclusion/exclusion and/or sensitivity analyses.	Graphical and tabular with narrative commentary.	Numerical analysis of measures of effect assuming absence of heterogeneity.
Mixed studies review/mixed methods review	Refers to any combination of methods where one significant component is a literature review (usually systematic). Within a review context it refers to a combination of review approaches for example combining quantitative with qualitative research or outcome with process studies.	Requires either very sensitive search to retrieve all studies or separately conceived quantitative and qualitative strategies.	Requires either a generic appraisal instrument or separate appraisal processes with corresponding checklists.	Typically both components will be presented as narrative and in tables. May also employ graphical means of integrating quantitative and qualitative studies.	Analysis may characterize both quantitative and qualitative studies and look for correlations between their characteristics or use gap analysis to identify aspects present in one type of study but missing in the other.
Overview	Generic term: summary of the [medical] literature that attempts to survey the literature and describe its characteristics.	May or may not include comprehensive searching (depends whether systematic overview or not).	May or may not include quality assessment (depends whether systematic overview or not).	Synthesis depends on whether systematic overview or not. Typically narrative but may include tabular features.	Analysis may be chronological, conceptual, thematic, etc.
Qualitative systematic review/qualitative evidence synthesis	Method for integrating or comparing the findings from qualitative studies. It looks for ‘themes’ or ‘constructs’ that lie in or across individual qualitative studies.	May employ selective or purposive sampling.	Quality assessment typically used to mediate messages not for inclusion/exclusion.	Qualitative, narrative synthesis.	Thematic analysis, may include conceptual models.
Rapid review	Assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research.	Completeness of searching determined by time constraints.	Time-limited formal quality assessment.	Typically narrative and tabular.	Quantities of literature and overall quality/direction of effect of literature.
Scoping review	Preliminary assessment of potential size and scope of available research literature. Aims to identify nature and extent of research evidence (usually including ongoing research).	Completeness of searching determined by time/scope constraints. May include research in progress.	No formal quality assessment.	Typically tabular with some narrative commentary.	Characterizes quantity and quality of literature, perhaps by study design and other key features. Attempts to specify a viable review.
State-of-the-art review	Tend to address more current matters in contrast to other combined retrospective and current approaches. May offer new perspectives on issue or point out area for further research.	Aims for comprehensive searching of current literature.	No formal quality assessment.	Typically narrative, may have tabular accompaniment.	Current state of knowledge and priorities for future investigation and research.
Systematic review	Seeks to systematically search for, appraise and synthesize research evidence, often adhering to guidelines on the conduct of a review.	Aims for exhaustive, comprehensive searching.	Quality assessment may determine inclusion/exclusion.	Typically narrative with tabular accompaniment.	What is known; recommendations for practice. What remains unknown; uncertainty around findings, recommendations for future research.
Systematic search and review	Combines strengths of critical review with a comprehensive search process. Typically addresses broad questions to produce ‘best evidence synthesis.'	Aims for exhaustive, comprehensive searching.	May or may not include quality assessment.	Minimal narrative, tabular summary of studies.	What is known; recommendations for practice. Limitations.
Systematized review	Attempt to include elements of systematic review process while stopping short of systematic review. Typically conducted as postgraduate student assignment.	May or may not include comprehensive searching.	May or may not include quality assessment.	Typically narrative with tabular accompaniment.	What is known; uncertainty around findings; limitations of methodology.
Umbrella review	Specifically refers to review compiling evidence from multiple reviews into one accessible and usable document. Focuses on broad condition or problem for which there are competing interventions and highlights reviews that address these interventions and their results.	Identification of component reviews, but no search for primary studies.	Quality assessment of studies within component reviews and/or of reviews themselves.	Graphical and tabular with narrative commentary.	What is known; recommendations for practice. What remains unknown; recommendations for future research.

Next: Writing the Protocol >>
Last Updated: Jul 22, 2024 3:29 PM
URL: https://guides.dml.georgetown.edu/systematicreviews

The Responsible Use of Electronic Resources policy governs the use of resources provided on these guides. © Dahlgren Memorial Library, Georgetown University Medical Center. Unless otherwise stated, these guides may be used for educational or academic purposes as long as proper attribution is given. Please seek permission for any modifications, adaptations, or for commercial purposes. Email [email protected] to request permission. Proper attribution includes: Written by or adapted from, Dahlgren Memorial Library, URL.

Harvey Cushing/John Hay Whitney Medical Library

Collections
Research Help

YSN Doctoral Programs: Steps in Conducting a Literature Review

Biomedical Databases
Global (Public Health) Databases
Soc. Sci., History, and Law Databases
Grey Literature
Trials Registers
Data and Statistics
Public Policy
Google Tips
Recommended Books
Steps in Conducting a Literature Review

What is a literature review?

A literature review is an integrated analysis -- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question. That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.

A literature review may be a stand alone work or the introduction to a larger research paper, depending on the assignment. Rely heavily on the guidelines your instructor has given you.

Why is it important?

A literature review is important because it:

Explains the background of research on a topic.
Demonstrates why a topic is significant to a subject area.
Discovers relationships between research studies/ideas.
Identifies major themes, concepts, and researchers on a topic.
Identifies critical gaps and points of disagreement.
Discusses further research questions that logically come out of the previous studies.

APA7 Style resources

APA Style Blog - for those harder to find answers

1. Choose a topic. Define your research question.

Your literature review should be guided by your central research question. The literature represents background and research developments related to a specific research question, interpreted and analyzed by you in a synthesized way.

Make sure your research question is not too broad or too narrow. Is it manageable?
Begin writing down terms that are related to your question. These will be useful for searches later.
If you have the opportunity, discuss your topic with your professor and your class mates.

2. Decide on the scope of your review

How many studies do you need to look at? How comprehensive should it be? How many years should it cover?

This may depend on your assignment. How many sources does the assignment require?

3. Select the databases you will use to conduct your searches.

Make a list of the databases you will search.

Where to find databases:

use the tabs on this guide
Find other databases in the Nursing Information Resources web page
More on the Medical Library web page
... and more on the Yale University Library web page

4. Conduct your searches to find the evidence. Keep track of your searches.

Use the key words in your question, as well as synonyms for those words, as terms in your search. Use the database tutorials for help.
Save the searches in the databases. This saves time when you want to redo, or modify, the searches. It is also helpful to use as a guide is the searches are not finding any useful results.
Review the abstracts of research studies carefully. This will save you time.
Use the bibliographies and references of research studies you find to locate others.
Check with your professor, or a subject expert in the field, if you are missing any key works in the field.
Ask your librarian for help at any time.
Use a citation manager, such as EndNote as the repository for your citations. See the EndNote tutorials for help.

Review the literature

Some questions to help you analyze the research:

What was the research question of the study you are reviewing? What were the authors trying to discover?
Was the research funded by a source that could influence the findings?
What were the research methodologies? Analyze its literature review, the samples and variables used, the results, and the conclusions.
Does the research seem to be complete? Could it have been conducted more soundly? What further questions does it raise?
If there are conflicting studies, why do you think that is?
How are the authors viewed in the field? Has this study been cited? If so, how has it been analyzed?

Tips:

Review the abstracts carefully.
Keep careful notes so that you may track your thought processes during the research process.
Create a matrix of the studies for easy analysis, and synthesis, across all of the studies.
<< Previous: Recommended Books
Last Updated: Jun 20, 2024 9:08 AM
URL: https://guides.library.yale.edu/YSNDoctoral

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

Advanced Search
Journal List
Can J Hosp Pharm
v.68(2); Mar-Apr 2015

Systematic Reviews: What Do You Need to Know to Get Started?

Introduction.

The cornerstone of clinical research on interventions is generally considered to be the randomized controlled trial (RCT). 1 However, in topic areas where the number of patients is limited or the evidence is conflicting, systematic reviews offer the benefit of collating evidence from a variety of sources. 1 , 2 A systematic review attempts to bring together all available evidence on a specific, clearly defined topic. Moreover, in areas where a number of large-scale trials have had similar results, a systematic review that includes meta-analysis of the data can help researchers to find a population estimate for the overall effect of the intervention.

In the hierarchy of evidence, systematic reviews of randomized trials offer the highest level of evidence. 1 The strongest inferences can be drawn if the systematic review is well conducted and includes methodologically sound RCTs with consistent results. In making treatment decisions, the highest quality of evidence should be sought, but well-conducted systematic reviews may not always be available.

This article provides a concise overview of the steps in a systematic review, with a focus on systematic reviews of RCTs. For more in-depth discussion of issues that go beyond the scope of this article, readers are directed to other resources. 2 , 3

DISTINCTION BETWEEN SYSTEMATIC REVIEWS AND META-ANALYSES

One of the most common errors in understanding systematic reviews is the perception that they are interchangeable with meta-analyses. “Systematic review” is the overarching term for studies that collate available evidence related to a directed clinical question. 2 , 4 A meta-analysis is a review in which statistical methods are employed to collate the numeric data from the primary studies. For various reasons, not all systematic reviews can combine the available data to generate summary numeric results; however, all systematic reviews should employ stringent methods to summarize the available research.

STUDY QUESTION

As in any type of research, a clear clinical question is needed for a systematic review. The question should state the patient group of interest, the intervention being investigated, the control or comparator group, and the outcomes of interest. 1 , 2 For example, if you were interested in conducting a systematic review of the effectiveness of new agents (e.g., gliptins) in the treatment of diabetes mellitus, you would need to clearly define your target population and specify whether you are considering studies that compare gliptins with other antidiabetic agents or studies that compare gliptins just with placebo. If you neglect to define each of these parameters a priori, you will have problems in determining which studies should be included and which should be excluded from your systematic review. Table 1 provides an example of how to define the question.

Example of Components in Posing a Relevant Clinical Question


Patient population	Patients with poorly controlled type 2 diabetes mellitus
Intervention	Gliptins (all agents)
Control	Placebo
Outcomes	Change in A1C, hypoglycemic episodes

A1C = glycated hemoglobin.

INCLUSION AND EXCLUSION CRITERIA

Just as an RCT requires that the investigator specify criteria for deciding which patients will be included in the study, a systematic review requires that the investigator define, at the proposal-writing stage, clear criteria for deciding which studies will be included. For example, will you limit your review to English-language studies? Will you include only RCTs, or will you accept other study designs? What outcomes will meet your inclusion criteria—will you include only hard clinical end points (e.g., admissions to hospital, deaths), or will you also consider surrogate end points (e.g., laboratory values)? The decisions about inclusion and exclusion criteria should relate directly to the defined research question. In the example given in Table 1 , you might want to consider what is meant by “poorly controlled” (e.g., defined in terms of glycated hemoglobin [A1C] or other clinical markers) and whether there is a specific age range you want to investigate (e.g., between 18 and 75 years). You might also decide whether studies must have a minimal methodologic quality to be considered for inclusion, particularly in terms of the study designs that you will consider acceptable, according to your area of study. RCTs represent the optimal study design; however, not all clinical questions can be answered by an RCT, so you may also need to include observational study designs, depending on the topic.

LITERATURE SEARCH

A systematic review is only as good as the data on which it is based, that is, the primary studies. To ensure that the widest scope of primary research is identified, a thorough and complete search of the literature is needed. The best way to accomplish this is to have help from a librarian with expertise in the area of systematic reviews in defining the search terms, search strategies, and databases to be used. The rule of thumb for a systematic review literature search is that more than 2 databases should be used. For the casual reader, the databases that should be used for a particular systematic review can be difficult to judge, but generally databases beyond just MEDLINE should be searched. Ideally, articles in languages other than English should be included, and there should be an attempt to find unpublished research and research that has not been formally published in a journal. This “grey literature” is the most difficult to find. Grey literature has been defined as “that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers”. 5 It can include reports, theses, conference proceedings, technical specifications and standards, noncommercial translations, bibliographies, technical and commercial documentation, and official documents not published commercially (such as government reports). 5 The instantaneous nature of the Internet has led to a proliferation of this type of literature, but the challenge lies in finding it. Searching the websites of agencies and organizations that may be involved in the area of interest is a reasonable way to start. Grey literature repositories and gateways have evolved and offer another method for researchers to find primary evidence not available in the peer-reviewed literature. 5 Again, searching for grey literature can be much easier with the help of a librarian.

In addition to a formal bibliographic search and a grey literature search, hand-searching of reference lists or key journals should be performed, as well as searching of controlled trial registers. This type of search can also be guided by a librarian. The difficulty with the search stage of a systematic review is keeping all of your potential studies organized. Online or shareable citation software is an easy way to manage the potentially large number of studies being considered. By using an online version, you can allow your co-researchers to have access to the studies. Given that at least 2 researchers will be needed to review the potentially included studies, online citation management allows for easy access by the multiple researchers.

STUDY SELECTION

At this stage, the eligibility of potential studies (based on the predefined inclusion and exclusion criteria) must be checked independently by at least 2 researchers. The lists of included studies prepared by the 2 reviewers are then compared and disagreements resolved, either through discussion or the involvement of a third researcher. The numbers of studies searched and included, and the reasons for study exclusion, should be recorded and presented as a flow diagram in the final manuscript. 6 Each and every potentially relevant study found through the search should be accounted for in this diagram.

DATA EXTRACTION AND QUALITY ASSESSMENT

As with study selection, data extraction should be conducted independently by 2 members of the research team. This helps to reduce or eliminate errors in the data compiled for analysis. Bias can also be reduced if the data extractors are blinded to study authors and journals of publication. To eliminate discrepancies, the data extraction forms should be standardized, so that the 2 reviewers are collecting the same information from each study. Similar to the process for study selection, a method for dispute resolution should be discussed a priori.

Determination of the quality of included studies should be based on whether the included studies have minimized bias in their study design (internal validity). The Cochrane Collaboration has moved away from using the term “quality” and now uses “bias” instead. 3 One reason for this shift is that focusing on bias allows you to concentrate on the quality of the underlying research, not just the reporting of that research. Meta-epidemiologic methods have been used to study the biases associated with study characteristics. 7 This type of work has led researchers to focus bias assessment on areas such as allocation concealment, randomization, blinding, and incompleteness of data and reporting. The Cochrane Collaboration has developed a tool to record whether these aspects of the study were done appropriately, not simply whether they were reported (as is commonly seen with checklist approaches to quality assessment). 3 A full description of the use and applicability of these tools is beyond the scope of this article; for more detailed instructions, readers are referred to the Cochrane Handbook for Systematic Reviews of Interventions (see Chapter 8: Assessing risk of bias in included studies). 3 The risk-of-bias tool is deemed more difficult and time-consuming to complete than previous methods of assessing bias; however, it is also viewed as providing more valuable information. 8 The main area of difficulty for researchers is how to incorporate risk-of-bias information into the data analysis. The Cochrane Collaboration is currently working on advanced instructions to assist in this domain. 8

Another step at this stage of the process is to contact the corresponding authors of the included studies for answers to any questions that have arisen during data extraction and for clarification of any areas of uncertainty in the methods and results. You can also inquire about any unpublished data that may contribute to your outcomes of interest. Authors are generally happy to provide this information, which helps to strengthen the results that you will generate.

DATA ANALYSIS

At this point in the review process, the hardest work has been completed, as you have already sifted through hundreds of citations and extracted data from each of the included studies. Now you need to determine if meta-analysis of the results will be appropriate. This decision will be driven mostly by whether or not the outcomes of the included studies were measured in a similar fashion and reported consistently. For example, if all of your included studies reported change in A1C at the end of the follow-up period, those data can easily be combined. However, if the studies report a variety of outcomes that are not directly comparable (e.g., A1C, fasting glucose, quality of life), the data cannot be combined. Other issues, such as similarity of patient populations, interventions, and methods used in the primary studies, will also need to be considered before data are combined; this is known as a qualitative assessment of heterogeneity. Although your study protocol should include your plan for data analysis based on your research question and outcomes of interest, you will not know the exact types of data you will collect until data extraction is complete.

The Cochrane Collaboration offers free software called Review Manager (RevMan) 9 that is useful for analyzing results and generating forest plots and risk-of-bias tables. Although this software is relatively straightforward and easy to use, you will need a basic understanding of statistical methods to make decisions on how the data should be analyzed. In addition, your primary outcome of interest and how it is measured (dichotomous versus continuous data) will affect the type of analysis you can do. For studies with dichotomous outcomes (e.g., dead or alive, admitted to hospital or not admitted), RevMan can calculate odds ratios and risk ratios from the included data. The generation of combined odds ratios (and confidence intervals) will also lead to an I 2 statistic to help in interpretation of heterogeneity (described in more detail below). How the software calculates the odds ratios will depend on your decision of what type of statistical model you want: fixed or random effects. A random-effects model assumes that the different studies are estimating different, yet related, intervention effects. 3 Generally, it is considered the most conservative method for meta-analyses. A complete discussion about determining what model should be used is beyond the scope of this article; however, the Cochrane Handbook for Systematic Reviews of Interventions 3 offers useful background information to help in answering some of these statistical questions. In addition, consultation with a statistician will help at this point of the research, if you have not already consulted one during proposal and study design. A statistician can also assist in selecting further statistical methods for subgroup analysis, sensitivity analysis, and investigation of sources of heterogeneity.

HETEROGENEITY

Heterogeneity is defined as any variability seen across the included studies. 3 For example, heterogeneity may derive from differences in study designs, populations, and outcomes. 10 If significant heterogeneity exists between the included studies, meta-analysis should not be considered. Even if the data can be combined, heterogeneity throughout the combined data should be measured. The most useful measure of heterogeneity is the I 2 statistic, which quantifies the level of heterogeneity present in each outcome and also helps in determining, post hoc, whether the data are interpretable. 10 A full discussion of heterogeneity is beyond the scope of this article, but readers can find more information in specific references on this topic. 3 , 10 These references also discuss methods that can be used to explore the sources of heterogeneity based on meta-regression and subgroup analyses.

INTERPRETATION AND REPORTING OF RESULTS

Now the work of the systematic review is done, and you want everyone to know what you found! However, putting the results together in a succinct and readable form can be difficult, especially if you have included a large volume of evidence.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement offers an outline of what should be included in a published systematic review. 6 The PRISMA checklist covers everything from what should be mentioned in the title, to what methods should be reported, to how the results should be presented and discussed. It is an excellent template for formatting your report and determining what may be missing (the PRISMA checklist can also be used as an aid during design of your study protocol, as it will help to frame some of the design decisions you need to make along the way). In general, the first figure in your article should be a clear flow diagram of all the studies found, the various exclusion steps, and the studies actually included in the systematic review. The first table in the systematic review should list all of the included studies, with details about each one, such as the patient population, intervention, control group, outcomes, and some basic results. Table 2 in the current article summarizes each step of the process of conducting a systematic review and will help you in drafting your final manuscript.

Key Steps in Conducting a Systematic Review


Define the clinical question	Use the PICO format (population, intervention, control, outcome).
Define inclusion and exclusion criteria	Consider study design, time of follow-up, clinical setting, comordities, demographic characteristics (e.g., age groups, sex), characteristics of the control group, outcome importance, language of publication, date of publication (as a range of years).
Conduct literature search	Consider appropriate databases, number of databases, publication date range, grey literature.
Select studies	Use at least 2 independent reviewers. Have the reviewers use the inclusion and exclusion criteria as the basis for their selections. Specify a method to resolve discrepancies (e.g., consensus, third reviewer).
Extract the data	Develop a data extraction form based on PICO. Have 2 researchers extract the data independently.
Determine level of bias (previously termed “study quality”) of included studies	The risk-of-bias tool is currently the preferred method. A different tool may be needed if non–randomized controlled trials are included.
Determine heterogeneity (qualitatively)	Determine whether there are major differences between studies in terms of patients, interventions, outcomes, study design (quality).
If appropriate, conduct meta-analysis	Using appropriate statistical software and the statistic, combine the data. Reassess outcomes for statistical heterogeneity. Generate summary estimates and confidence intervals.
Evaluate and interpret results	Consider clinical significance and practice context.
Publish	Follow the PRISMA guidelines in drafting your manuscript.

PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

To ensure you are producing high-level evidence, begin by partnering with researchers who have experience with systematic reviews and ensuring accessibility to a librarian and a statistician. Using available resources, such as the Cochrane Collaboration and the PRISMA guidelines, develop a strong question and project proposal, with clearly defined inclusion and exclusion criteria, literature search strategy, and data analysis plan. Do not be discouraged if the data cannot be analyzed using meta-analytic methods, as the systematic review can still be highly relevant and useful without pooled data. By following these steps, you should be able to develop a plan and assemble a strong team to move your planned review forward.

This article is the ninth in the CJHP Research Primer Series, an initiative of the CJHP Editorial Board and the CSHP Research Committee. The planned 2-year series is intended to appeal to relatively inexperienced researchers, with the goal of building research capacity among practising pharmacists. The articles, presenting simple but rigorous guidance to encourage and support novice researchers, are being solicited from authors with appropriate expertise.

Previous article in this series:

Bond CM. The research jigsaw: how to get started. Can J Hosp Pharm . 2014;67(1):28–30.

Tully MP. Research: articulating questions, generating hypotheses, and choosing study designs. Can J Hosp Pharm . 2014;67(1):31–4.

Loewen P. Ethical issues in pharmacy practice research: an introductory guide. Can J Hosp Pharm. 2014;67(2):133–7.

Tsuyuki RT. Designing pharmacy practice research trials. Can J Hosp Pharm . 2014;67(3):226–9.

Bresee LC. An introduction to developing surveys for pharmacy practice research. Can J Hosp Pharm . 2014;67(4):286–91.

Gamble JM. An introduction to the fundamentals of cohort and case–control studies. Can J Hosp Pharm . 2014;67(5):366–72.

Austin Z, Sutton J. Qualitative research: getting started. C an J Hosp Pharm . 2014;67(6):436–40.

Houle S. An introduction to the fundamentals of randomized controlled trials in pharmacy research. Can J Hosp Pharm . 2014; 68(1):28–32.

Competing interests: None declared.

IMAGES

How to Conduct a Systematic Review
Systematic Literature Review Methodology
Systematic literature review phases.
Why do we need systematic literature reviews?
Why and how to conduct a systematic literature review
Systematic literature review phases.

COMMENTS

Why systematic reviews matter
A potent means of doing so is that of systematic reviews. The continuous growth of research, coupled with the demand to systematically summarize the available evidence to inform decisions from consumers and stakeholders, led to the formal development of systematic reviews (SRs) in the late 20th century 1-3.
Guidance on Conducting a Systematic Literature Review
Literature reviews establish the foundation of academic inquires. However, in the planning field, we lack rigorous systematic reviews. In this article, through a systematic search on the methodology of literature review, we categorize a typology of literature reviews, discuss steps in conducting a systematic literature review, and provide suggestions on how to enhance rigor in literature ...
Chapter 1: Starting a review
1.1 Why do a systematic review? Systematic reviews were developed out of a need to ensure that decisions affecting people's lives can be informed by an up-to-date and complete understanding of the relevant research evidence. With the volume of research literature growing at an ever-increasing rate, it is impossible for individual decision ...
Systematic Reviews and Meta-Analysis: A Guide for Beginners
Systematic reviews involve the application of scientific methods to reduce bias in review of literature. The key components of a systematic review are a well-defined research question, comprehensive literature search to identify all studies that potentially address the question, systematic assembly of the studies that answer the question, critical appraisal of the methodological quality of the ...
How-to conduct a systematic literature review: A quick guide for
Method details Overview. A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12].An SLR updates the reader with current literature about a subject [6].The goal is to review critical points of current knowledge on a ...
Systematic reviews: Brief overview of methods, limitations, and
CONCLUSION. Siddaway 16 noted that, "The best reviews synthesize studies to draw broad theoretical conclusions about what the literature means, linking theory to evidence and evidence to theory" (p. 747). To that end, high quality systematic reviews are explicit, rigorous, and reproducible. It is these three criteria that should guide authors seeking to write a systematic review or editors ...
How to Do a Systematic Review: A Best Practice Guide for Conducting and
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question.
Systematic reviews: Structure, form and content
Abstract. This article aims to provide an overview of the structure, form and content of systematic reviews. It focuses in particular on the literature searching component, and covers systematic database searching techniques, searching for grey literature and the importance of librarian involvement in the search.
How to do a systematic review
A systematic review aims to bring evidence together to answer a pre-defined research question. This involves the identification of all primary research relevant to the defined review question, the critical appraisal of this research, and the synthesis of the findings.13 Systematic reviews may combine data from different.
Systematically Reviewing the Literature: Building the Evidence for
Introduction. Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review. It is important to systematically review the literature when one would like to justify the need for a study, to update ...
Systematic Review
A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. ... Systematic review vs. literature review. A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize ...
How-to conduct a systematic literature review: A quick guide for
Abstract. Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in ...
Guidelines for writing a systematic review
A preliminary review, which can often result in a full systematic review, to understand the available research literature, is usually time or scope limited. Complies evidence from multiple reviews and does not search for primary studies. 3. Identifying a topic and developing inclusion/exclusion criteria.
Easy guide to conducting a systematic review
A systematic review is a type of study that synthesises research that has been conducted on a particular topic. Systematic reviews are considered to provide the highest level of evidence on the hierarchy of evidence pyramid. Systematic reviews are conducted following rigorous research methodology. To minimise bias, systematic reviews utilise a ...
How to do a systematic review
A systematic review aims to bring evidence together to answer a pre-defined research question. This involves the identification of all primary research relevant to the defined review question, the critical appraisal of this research, and the synthesis of the findings. 13 Systematic reviews may combine data from different research studies in order to produce a new integrated result or ...
Literature review as a research methodology: An ...
2.1.1. Systematic literature review. What is it and when should we use it? Systematic reviews have foremost been developed within medical science as a way to synthesize research findings in a systematic, transparent, and reproducible way and have been referred to as the gold standard among reviews (Davis et al., 2014).Despite all the advantages of this method, its use has not been overly ...
How to Do a Systematic Review: A Best Practice Guide for Conducting and
Systematic reviews are characterized by a methodical and replicable methodology and presentation. They involve a comprehensive search to locate all relevant published and unpublished work on a subject; a systematic integration of search results; and a critique of the extent, nature, and quality of evidence in relation to a particular research question. The best reviews synthesize studies to ...
Systematic Reviews and Meta-analysis: Understanding the Best Evidence
Systematic review. A systematic review is a summary of the medical literature that uses explicit and reproducible methods to systematically search, critically appraise, and synthesize on a specific issue. It synthesizes the results of multiple primary studies related to each other by using strategies that reduce biases and random errors.
Guides: Systematic Reviews: Should I do a systematic review?
A systematic literature review is a research methodology designed to answer a focused research question. Authors conduct a methodical and comprehensive literature synthesis focused on a well-formulated research question. Its aim is to identify and synthesize all of the scholarly research on a particular topic, including both published and ...
Five steps to conducting a systematic review
A review earns the adjective systematic if it is based on a clearly formulated question, identifies relevant studies, appraises their quality and summarizes the evidence by use of explicit methodology. It is the explicit and systematic approach that distinguishes systematic reviews from traditional reviews and commentaries.
Steps in Conducting a Literature Review
A literature review is an integrated analysis-- not just a summary-- of scholarly writings and other relevant evidence related directly to your research question. That is, it represents a synthesis of the evidence that provides background information on your topic and shows a association between the evidence and your research question.
Systematic Reviews: What Do You Need to Know to Get Started?
LITERATURE SEARCH. A systematic review is only as good as the data on which it is based, that is, the primary studies. To ensure that the widest scope of primary research is identified, a thorough and complete search of the literature is needed. The best way to accomplish this is to have help from a librarian with expertise in the area of ...