chapter 3 research methods reliability fill in the gaps

• Welcome to Chapter 3

Chapter 3 Webinars

• Student Experience Feedback Buttons
• Developing the Quantitative Research Design
• Qualitative Descriptive Design
• Qualitative Narrative Inquiry Research
• SAGE Research Methods
• Alignment of Dissertation Components for DIS-9902ABC
• IRB Resources This link opens in a new window
• Research Examples (SAGE) This link opens in a new window
• Dataset Examples (SAGE) This link opens in a new window

Jump to DSE Guide

Need help ask us.

Was this resource helpful?

• Next: Developing the Quantitative Research Design >>
• Last Updated: Nov 2, 2023 10:17 AM
• URL: https://resources.nu.edu/c.php?g=1007179

© Copyright 2024 National University. All Rights Reserved.

Privacy Policy | Consumer Information

Components of Research Methodology

• First Online: 01 March 2024

Cite this chapter

• Animesh Hazari 2  

286 Accesses

This chapter gives an overall understanding of components used in any research sequentially. These components would be followed and used for conducting as well as writing the research projects. In my personal experience, it would lead the readers to conduct their research projects with accuracy and scientific rigor. Here, we will try to list the components and briefly discuss them. Detailed information will be given in the chapters ahead. These components of research methodology can vary depending on the field of study and the research design, but here are the key components commonly found in most research methodologies:

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Author information

Authors and affiliations.

College of Health Sciences, Gulf Medical University, Ajman, Ajman, United Arab Emirates

Animesh Hazari

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this chapter

Hazari, A. (2023). Components of Research Methodology. In: Research Methodology for Allied Health Professionals. Springer, Singapore. https://doi.org/10.1007/978-981-99-8925-6_3

Download citation

DOI : https://doi.org/10.1007/978-981-99-8925-6_3

Published : 01 March 2024

Publisher Name : Springer, Singapore

Print ISBN : 978-981-99-8924-9

Online ISBN : 978-981-99-8925-6

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Gen Intern Med
• v.37(1); 2022 Jan

Methods for Identifying Health Research Gaps, Needs, and Priorities: a Scoping Review

Eunice c. wong.

1 RAND Corporation, Santa Monica, CA USA

Alicia R. Maher

Aneesa motala.

2 Department of Population and Public Health Sciences, University of Southern California Gehr Family Center for Health Systems Science & Innovation, Los Angeles, USA

Rachel Ross

Olamigoke akinniranye, jody larkin, susanne hempel, associated data.

Well-defined, systematic, and transparent processes to identify health research gaps, needs, and priorities are vital to ensuring that available funds target areas with the greatest potential for impact.

The purpose of this review is to characterize methods conducted or supported by research funding organizations to identify health research gaps, needs, or priorities.

We searched MEDLINE, PsycINFO, and the Web of Science up to September 2019. Eligible studies reported on methods to identify health research gaps, needs, and priorities that had been conducted or supported by research funding organizations. Using a published protocol, we extracted data on the method, criteria, involvement of stakeholders, evaluations, and whether the method had been replicated (i.e., used in other studies).

Among 10,832 citations, 167 studies were eligible for full data extraction. More than half of the studies employed methods to identify both needs and priorities, whereas about a quarter of studies focused singularly on identifying gaps (7%), needs (6%), or priorities (14%) only. The most frequently used methods were the convening of workshops or meetings (37%), quantitative methods (32%), and the James Lind Alliance approach, a multi-stakeholder research needs and priority setting process (28%). The most widely applied criteria were importance to stakeholders (72%), potential value (29%), and feasibility (18%). Stakeholder involvement was most prominent among clinicians (69%), researchers (66%), and patients and the public (59%). Stakeholders were identified through stakeholder organizations (51%) and purposive (26%) and convenience sampling (11%). Only 4% of studies evaluated the effectiveness of the methods and 37% employed methods that were reproducible and used in other studies.

To ensure optimal targeting of funds to meet the greatest areas of need and maximize outcomes, a much more robust evidence base is needed to ascertain the effectiveness of methods used to identify research gaps, needs, and priorities.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11606-021-07064-1.

Well-defined, systematic, and transparent methods to identify health research gaps, needs, and priorities are vital to ensuring that available funds target areas with the greatest potential for impact. 1 , 2 As defined in the literature, 3 , 4 research gaps are defined as areas or topics in which the ability to draw a conclusion for a given question is prevented by insufficient evidence. Research gaps are not necessarily synonymous with research needs , which are those knowledge gaps that significantly inhibit the decision-making ability of key stakeholders, who are end users of research, such as patients, clinicians, and policy makers. The selection of research priorities is often necessary when all identified research gaps or needs cannot be pursued because of resource constraints. Methods to identify health research gaps, needs, and priorities (from herein referred to as gaps, needs, priorities) can be multi-varied and there does not appear to be general consensus on best practices. 3 , 5

Several published reviews highlight the diverse methods that have been used to identify gaps and priorities. In a review of methods used to identify gaps from systematic reviews, Robinson et al. noted the wide range of organizing principles that were employed in published literature between 2001 and 2009 (e.g., care pathway, decision tree, and patient, intervention, comparison, outcome framework,). 6 In a more recent review spanning 2007 to 2017, Nyanchoka et al. found that the vast majority of studies with a primary focus on the identification of gaps (83%) relied solely on knowledge synthesis methods (e.g., systematic review, scoping review, evidence mapping, literature review). A much smaller proportion (9%) relied exclusively on primary research methods (i.e., quantitative survey, qualitative study). 7

With respect to research priorities, in a review limited to a PubMed database search covering the period from 2001 to 2014, Yoshida documented a wide range of methods to identify priorities including the use of not only knowledge synthesis (i.e., literature reviews) and primary research methods (i.e., surveys) but also multi-stage, structured methods such as Delphi, Child Health and Nutrition Research Initiative (CHNRI), James Lind Alliance Priority Setting Partnership (JLA PSP), and Essential National Health Research (ENHR). 2 The CHNRI method, originally developed for the purpose of setting global child health research priorities, typically employs researchers and experts to specify a long list of research questions, the criteria that will be used to prioritize research questions, and the technical scoring of research questions using the defined criteria. 8 During the latter stages, non-expert stakeholders’ input are incorporated by using their ratings of the importance of selected criteria to weight the technical scores. The ENHR method, initially designed for health research priority setting at the national level, involves researchers, decision-makers, health service providers, and communities throughout the entire process of identifying and prioritizing research topics. 9 The JLA PSP method convenes patients, carers, and clinicians to equally and jointly identify questions about healthcare that cannot be answered by existing evidence that are important to all groups (i.e., research needs). 10 The identified research needs are then prioritized by the groups resulting in a final list (often a top 10) of research priorities. Non-clinical researchers are excluded from voting on research needs or priorities but can be involved in other processes (e.g., knowledge synthesis). CHNRI, ENHR, and JLA PSP usually employ a mix of knowledge synthesis and primary research methods to first identify a set of gaps or needs that are then prioritized. Thus, even though CHNRI, ENHR, and JLA PSP have been referred to as priority setting methods, they actually consist of a gaps or needs identification stage that feeds into a research prioritization stage.

Nyanchoka et al.’s review found that the majority of studies focused on the identification of gaps alone (65%), whereas the remaining studies focused either on research priorities alone (17%) or on both gaps and priorities (19%). 7 In an update to Robinson et al.’s review, 6 Carey et al. reviewed the literature between 2010 and 2011 and observed that the studies conducted during this latter period of time focused more on research priorities than gaps and had increased stakeholder involvement, and that none had evaluated the reproducibility of the methods. 11

The increasing development and diversity of formal processes and methods to identify gaps and priorities are indicative of a developing field. 2 , 12 To facilitate more standardized and systematic processes, other important areas warrant further investigation. Prior reviews did not distinguish between the identification of gaps versus research needs. The Agency for Healthcare Research and Quality Evidence-based Practice Center (AHRQ EPC) Program issued a series of method papers related to establishing research needs as part of comparative effectiveness research. 13 – 15 The AHRQ EPC Program defined research needs as “evidence gaps” identified within systematic reviews that are prioritized by stakeholders according to their potential impact on practice or care. 16 Furthermore, Nyanchoka et al. relied on author designations to classify studies as focusing on gaps versus research priorities and noted that definitions of gaps varied across studies, highlighting the need to apply consistent taxonomy when categorizing studies in reviews. 7 Given the rise in the use of stakeholders in both gaps and prioritization exercises, a greater understanding of the range of practices involving stakeholders is also needed. This includes the roles and responsibilities of stakeholders (e.g., consultants versus final decision-makers), the composition of stakeholders (e.g., non-research clinicians, patients, caregivers, policymakers), and the methods used to recruit stakeholders. The lack of consensus of best practices also highlights the importance of learning the extent to which evaluations to determine the effectiveness of gaps, needs, and prioritization exercises have been conducted, and if so, what were the resultant outcomes.

To better inform efforts and organizations that fund health research, we conducted a scoping review of methods used to identify gaps, needs, and priorities that were linked to potential or actual health research funding decision-making. Hence, this scoping review was limited to studies in which the identification of health research gaps, needs, or priorities was supported or conducted by funding organizations to address the following questions 1 : What are the characteristics of methods to identify health research gaps, needs, and priorities? and 2 To what extent have evaluations of the impact of these methods been conducted? Given that scoping reviews may be executed to characterize the ways an area of research has been conducted, 17 , 18 this approach is appropriate for the broad nature of this study’s aims.

Protocol and Registration

We employed methods that conform to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews. 19 See Appendix A in the Supplementary Information. The scoping review protocol is registered with the Open Science Framework ( https://osf.io/5zjqx/ ).

Eligibility Criteria

Studies published in English that described methods to identify health research gaps, needs, or priorities that were supported or conducted by funding organizations were eligible for inclusion. We excluded studies that reported only the results of the exercise (e.g., list of priorities) absent of information on the methods used. We also excluded studies involving evidence synthesis (e.g., literature or systematic reviews) that were solely descriptive and did not employ an explicit method to identify research gaps, needs, or priorities.

Information Sources and Search Strategy

We searched the following electronic databases: MEDLINE, PsycINFO, and Web of Science. Our database search also included an update of the Nyanchoka et al. scoping review, which entailed executing their database searches for the time period following 2017 (the study’s search end date). 7 Nyanchoka et al. did not include database searches for research needs. The electronic database search and scoping review update were completed in August and September 2019, respectively . The search strategy employed for each of the databases is presented in Appendix B in the Supplementary Information.

Selection of Sources of Evidence and Data Charting Process

Two reviewers screened titles and abstracts and full-text publications. Citations that one or both reviewers considered potentially eligible were retrieved for full-text review. Relevant background articles and scoping and systematic reviews were reference mined to screen for eligible studies. Full-text publications were screened against detailed inclusion and exclusion criteria. Data was extracted by one reviewer and checked by a second reviewer. Discrepancies were resolved through discussion by the review team.

Information on study characteristics were extracted from each article including the aims of the exercise (i.e., gaps, needs, priorities, or a combination) and health condition (i.e., physical or psychological). Based on definitions in the literature, 3 – 5 the aims of the exercise were coded according to the activities that were conducted, which may not have always corresponded with the study authors’ labeling of the exercises. For instance, the JLA PSP method is often described as a priority exercise but we categorized it as a needs and priority exercise. Priority exercises can be preceded by exercises to identify gaps or needs, which then feed into the priority exercise such as in JLA PSP; however, standalone priority exercises can also be conducted (e.g., stakeholders prioritize an existing list of emerging diseases).

For each type of exercise, information on the methods were recorded. An initial list of methods was created based on previous reviews. 9 , 12 , 20 During the data extraction process, any methods not included in the initial list were subsequently added. If more than one exercise was reported within an article (e.g., gaps and priorities), information was extracted for each exercise separately. Reviewers extracted the following information: methods employed (e.g., qualitative, quantitative), criteria used (e.g., disease burden, importance to stakeholders), stakeholder involvement (e.g., stakeholder composition, method for identifying stakeholders), and whether an evaluation was conducted on the effectiveness of the exercise (see Appendix C in the Supplementary Information for full data extraction form).

Synthesis of results entailed quantitative descriptives of study characteristics (e.g., proportion of studies by aims of exercise) and characteristics of methods employed across all studies and by each type of study (e.g., gaps, needs, priorities).

The electronic database search yielded a total of 10,548 titles. Another 284 articles were identified after searching the reference lists of full-text publications, including three systematic reviews 21 – 23 and one scoping review 24 that had met eligibility criteria. Moreover, a total of 99 publications designated as relevant background articles were also reference mined to screen for eligible studies. We conducted full-text screening for 2524 articles, which resulted in 2344 exclusions (440 studies were designated as background articles). A total of 167 exercises related to the identification of gaps, needs, or priorities that were supported or conducted by a research funding organization were described across 180 publications and underwent full data extraction. See Figure ​ Figure1 1 for the flow diagram of our search strategy and reasons for exclusion.

Literature flow

Characteristics of Sources of Evidence

Among the published exercises, the majority of studies (152/167) conducted gaps, need, or prioritization exercises related to physical health, whereas only a small fraction of studies focused on psychological health (12/167) (see Appendix D in the Supplementary Information).

Methods for Identifying Gaps, Needs, and Priorities

As seen in Table ​ Table1, 1 , only about a quarter of studies involved a singular type of exercise with 7% focused on the identification of gaps only (i.e., areas with insufficient information to draw a conclusion for a given question), 6% on needs only (i.e., knowledge gaps that inhibit the decision-making of key stakeholders), and 14% priorities only (i.e., ranked gaps or needs often because of resource constraints). Studies more commonly conducted a combination of multiple types of exercises with more than half focused on the identification of both research needs and priorities, 14% on gaps and priorities, 3% gaps, needs, and priorities, and 3% gaps and needs.

Methods for Identifying Health Research Gaps, Needs, and Priorities

JLA PSP , James Lind Alliance Priority Setting Partnerships; ENHR , Essential National Health Research; CHNRI , Child Health and Nutrition Research Initiative. Numbers in columns may add up to more than the total N or 100% since some studies employed more than one method

Across the 167 studies, the three most frequently used methods were the convening of workshops/meetings/conferences (37%), quantitative methods (32%), and the JLA PSP approach (28%). This was followed by methods involving literature reviews (17%), qualitative methods (17%), consensus methods (13%), and reviews of source materials (15%). Other methods included the CHNRI process (7%), reviews of in-progress data (7%), consultation with (non-researcher) stakeholders (4%), applying a framework tool (4%), ENHR (1%), systematic reviews (1%), and evidence mapping (1%).

The criterion most widely applied across the 167 studies was the importance to stakeholders (72%) (see Table ​ Table2). 2 ). Almost one-third (29%) considered the potential value and 18% feasibility as criteria. Burden of disease (9%), addressing inequities (8%), costs (6%), alignment with organization’s mission (3%), and patient centeredness (2%) were adopted as criteria to a lesser extent.

Criteria for Identifying Health Research Gaps, Needs, and Priorities

Numbers in columns may add up to more than the total N or 100% since some studies employed more than one criterion

About two-thirds of the studies included researchers (66%) and clinicians (69%) as stakeholders (see Appendix E in the Supplementary Information). Patients and the public were involved in 59% of the studies. A smaller proportion included policy makers (20%), funders (13%), product makers (8%), payers (5%), and purchasers (2%) as stakeholders. Nearly half of the studies (51%) relied on stakeholder organizations to identify stakeholders (see Appendix F in the Supplementary Information). A quarter of studies (26%) used purposive sampling and some convenience sampling (11%). Few (9%) used snowball sampling to identify stakeholders. Only a minor fraction of studies, seven of the 167 (4%), reported some type of effectiveness evaluation. 25 – 31

Our scoping review revealed that approaches to identifying gaps, needs, and priorities are less likely to occur as discrete processes and more often involve a combination of exercises. Approaches encompassing multiple exercises (e.g., gaps and needs) were far more prevalent than singular standalone exercises (e.g., gaps only) (73% vs. 27%). Findings underscore the varying importance placed on gaps, needs, and priorities, which reflect key principles of the Value of Information approach (i.e., not all gaps are important, addressing gaps do not necessarily address needs nor does addressing needs necessarily address priorities). 32

Findings differ from Nyanchoka et al.’s review in which studies involving the identification of gaps only outnumbered studies involving both gaps and priorities. 7 However, Nyanchoka et al. relied on author definitions to categorize exercises, whereas our study made designations based on our review of the activities described in the article and applied definitions drawn from the literature. 3 , 4 Lack of consensus on definitions of gaps and priority setting has been noted in the literature. 33 , 34 To the authors’ knowledge, no prior scoping review has focused on methods related to the identification of “research needs.” Findings underscore the need to develop and apply more consistent taxonomy to this growing field of research.

More than 40% of studies employed methods with a structured protocol including JLA PSP, ENHR, CHRNI, World Café, and the Dialogue model. 10 , 35 – 40 The World Café and Dialogue models particularly value the experiential perspectives of stakeholders. The World Café centers on creating a special environment, often modeled after a café, in which rounds of multi-stakeholder, small group, conversations are facilitated and prefaced with questions designed for the specific purpose of the session. Insights and results are reported and shared back to the entire group with no expectation to achieve consensus, but rather diverse perspectives are encouraged. 36 The Dialogue model is a multi-stakeholder, participatory, priority setting method involving the following phases: exploratory (informal discussions), consultation (separate stakeholder consultations), prioritization (stakeholder ratings), and integration (dialog between stakeholders). 39 Findings may indicate a trend away from non-replicable methods to approaches that afford greater transparency and reproducibility. 41 For instance, of the 17 studies published between 2000 and 2009, none had employed CHNRI and 6% used JLA PSP compared to the 141 studies between 2010 and 2019 in which 8% applied CHNRI and 32% JLA PSP. However, notable variations in implementing CHNRI and JLA PSP have been observed. 41 – 43 Though these protocols help to ensure a more standardized process, which is essential when testing the effectiveness of methods, such evaluations are infrequent but necessary to establish the usefulness of replicable methods.

Convening workshops, meetings, or conferences was the method used by the greatest proportion of studies (37%). The operationalization of even this singular method varied widely in duration (e.g., single vs. multi-day conferences), format (e.g., expert panel presentations, breakout discussion groups), processes (e.g., use of formal/informal consensus methods), and composition of stakeholders. The operationalization of other methods (e.g., quantitative, qualitative) also exhibited great diversity.

The use of explicit criteria to determine gaps, needs, or priorities is a key component of certain structured protocols 40 , 44 and frameworks. 9 , 45 In our scoping review, the criterion applied most frequently across studies (71%) was “importance to stakeholders” followed by potential value (31%) and feasibility (18%). Stakeholder values are being incorporated into the identification of gaps, needs, and exercises across a significant proportion of studies, but how this is operationalized varies widely across studies. For instance, the CHNRI typically employs multiple criteria that are scored by technical experts and these scores are then weighted based on stakeholder ratings of their relative importance. Other studies totaled scores across multiple criteria, whereas JLA PSP asks multiple stakeholders to rank the top ten priorities. The importance of involving stakeholders, especially patients and the public, in priority setting is increasingly viewed as vital to ensuring the needs of end users are met, 46 , 47 particularly in light of evidence demonstrating mismatches between the research interests of patients and researchers and clinicians. 48 – 50 In our review, clinicians (69%) and researchers (66%) were the most widely represented stakeholder groups across studies. Patients and the public (e.g., caregivers) were included as stakeholders in 59% of the studies. Only a small fraction of studies involved exercises in which stakeholders were limited to researchers only. Patients and the public were involved as stakeholders in 12% of studies published between 2000 and 2009 compared to 60% of studies between 2010 and 2019. Findings may reflect a trend away from researchers traditionally serving as one of the sole drivers of determining which research topics should be pursued.

More than half of the studies reported relying on stakeholder organizations to identify participants. Partnering with stakeholder organizations has been noted as one of the primary methods for identifying stakeholders for priority setting exercises. 34 Purposive sampling was the next most frequently used stakeholder identification method. In contrast, convenience sampling (e.g., recommendations by study team) and snowball sampling (e.g., identified stakeholders refer other stakeholders who then refer additional stakeholders) were not as frequently employed, but were documented as common methods in a prior review conducted almost a decade ago. 14 The greater use of stakeholder organizations than convenience or snowball sampling may be partly due to the more recent proliferation of published studies using structured protocols like JLA PSP, which rely heavily on partnerships with stakeholder organizations. Though methods such as snowball sampling may introduce more bias than random sampling, 14 there are no established best practices for stakeholder identification methods. 51 Nearly a quarter of studies provided either unclear or no information on stakeholder identification methods, which has been documented as a barrier to comparing across studies and assessing the validity of research priorities. 34

Determining the effectiveness of gaps, needs, and priority exercises is challenging given that outcome evaluations are rarely conducted. Only seven studies reported conducting an evaluation. 25 – 31 Evaluations varied with respect to their focus on process- (e.g., balanced stakeholder representation, stakeholder satisfaction) versus outcome-related impact (e.g., prioritized topics funded, knowledge production, benefits to health). There is no consensus on what constitutes optimal outcomes, which has been found to vary by discipline. 52

More than 90% of studies involved exercises related to physical health in contrast to a minor portfolio of work being dedicated to psychological health, which may be an indication of the low priority placed on psychological health policy research. Understanding whether funding decisions for physical versus psychological health research are similarly or differentially governed by more systematic, formal processes may be important to the extent that this affects the effective targeting of funds.

Limitations

By limiting studies to those supported or conducted by funding organizations, we may have excluded global, national, or local priority setting exercises. In addition, our scoping review categorized approaches according to the actual exercises conducted and definitions provided in the scientific literature rather than relying on the terminology employed by studies. This resulted in instances in which the category assigned to an exercise within our scoping review could diverge from the category employed by the study authors. Lastly, this study’s findings are subject to limitations often characteristic of scoping reviews such as publication bias, language bias, lack of quality assessment, and search, inclusion, and extraction biases. 53

Conclusions

The diversity and growing establishment of formal processes and methods to identify health research gaps, needs, and priorities are characteristic of a developing field. Even with the emergence of more structured and systematic approaches, the inconsistent categorization and definition of gaps, needs, and priorities inhibit efforts to evaluate the effectiveness of varied methods and processes, such efforts are rare and sorely needed to build an evidence base to guide best practices. The immense variation occurring within structured protocols, across different combinations of disparate methods, and even within singular methods, further emphasizes the importance of using clearly defined approaches, which are essential to conducting investigations of the effectiveness of these varied approaches. The recent development of reporting guidelines for priority setting for health research may facilitate more consistent and clear documentation of processes and methods, which includes the many facets of involving stakeholders. 34 To ensure optimal targeting of funds to meet the greatest areas of need and maximize outcomes, a much more robust evidence base is needed to ascertain the effectiveness of methods used to identify research gaps, needs, and priorities.

(PDF 1205 kb)

Acknowledgements

This scoping review is part of research that was sponsored by Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury (now Psychological Health Center of Excellence).

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.