mixed method research articles

Mixed methods research: what it is and what it could be

• Open access
• Published: 29 March 2019
• Volume 48 , pages 193–216, ( 2019 )

Cite this article

You have full access to this open access article

• Rob Timans 1 ,
• Paul Wouters 2 &
• Johan Heilbron 3  

127k Accesses

107 Citations

12 Altmetric

Explore all metrics

A Correction to this article was published on 06 May 2019

This article has been updated

Combining methods in social scientific research has recently gained momentum through a research strand called Mixed Methods Research (MMR). This approach, which explicitly aims to offer a framework for combining methods, has rapidly spread through the social and behavioural sciences, and this article offers an analysis of the approach from a field theoretical perspective. After a brief outline of the MMR program, we ask how its recent rise can be understood. We then delve deeper into some of the specific elements that constitute the MMR approach, and we engage critically with the assumptions that underlay this particular conception of using multiple methods. We conclude by offering an alternative view regarding methods and method use.

Similar content being viewed by others

Mixed Methods

Explore related subjects

• Artificial Intelligence

Avoid common mistakes on your manuscript.

The interest in combining methods in social scientific research has a long history. Terms such as “triangulation,” “combining methods,” and “multiple methods” have been around for quite a while to designate using different methods of data analysis in empirical studies. However, this practice has gained new momentum through a research strand that has recently emerged and that explicitly aims to offer a framework for combining methods. This approach, which goes by the name of Mixed Methods Research (MMR), has rapidly become popular in the social and behavioural sciences. This can be seen, for instance, in Fig.  1 , where the number of publications mentioning “mixed methods” in the title or abstract in the Thomson Reuters Web of Science is depicted. The number increased rapidly over the past ten years, especially after 2006. Footnote 1

Fraction of the total of articles mentioning Mixed Method Research appearing in a given year, 1990–2017 (yearly values sum to 1). See footnote 1

The subject of mixed methods thus seems to have gained recognition among social scientists. The rapid rise of the number of articles mentioning the term raises various sociological questions. In this article, we address three of these questions. The first question concerns the degree to which the approach of MMR has become institutionalized within the field of the social sciences. Has MMR become a recognizable realm of knowledge production? Has its ascendance been accompanied by the production of textbooks, the founding of journals, and other indicators of institutionalization? The answer to this question provides an assessment of the current state of MMR. Once that is determined, the second question is how MMR’s rise can be understood. Where does the approach come from and how can its emergence and spread be understood? To answer this question, we use Pierre Bourdieu’s field analytical approach to science and academic institutions (Bourdieu 1975 , 1988 , 2004 , 2007 ; Bourdieu et al. 1991 ). We flesh out this approach in the next section. The third question concerns the substance of the MMR corpus seen in the light of the answers to the previous questions: how can we interpret the specific content of this approach in the context of its socio-historical genesis and institutionalization, and how can we understand its proposal for “mixing methods” in practice?

We proceed as follows. In the next section, we give an account of our theoretical approach. Then, in the third, we assess the degree of institutionalization of MMR, drawing on the indicators of academic institutionalization developed by Fleck et al. ( 2016 ). In the fourth section, we address the second question by examining the position of the academic entrepreneurs behind the rise of MMR. The aim is to understand these agents’ engagement in MMR, as well as its distinctive content as being informed by their position in this field. Viewing MMR as a position-taking of academic entrepreneurs, linked to their objective position in this field, allows us to reflect sociologically on the substance of the approach. We offer this reflection in the fifth section, where we indicate some problems with MMR. To get ahead of the discussion, these problems have to do with the framing of MMR as a distinct methodology and its specific conceptualization of data and methods of data analysis. We argue that these problems hinder fruitfully combining methods in a practical understanding of social scientific research. Finally, we conclude with some tentative proposals for an alternative view on combining methods.

A field approach

Our investigation of the rise and institutionalization of MMR relies on Bourdieu’s field approach. In general, field theory provides a model for the structural dimensions of practices. In fields, agents occupy a position relative to each other based on the differences in the volume and structure of their capital holdings. Capital can be seen as a resource that agents employ to exert power in the field. The distribution of the form of capital that is specific to the field serves as a principle of hierarchization in the field, differentiating those that hold more capital from those that hold less. This principle allows us to make a distinction between, respectively, the dominant and dominated factions in a field. However, in mature fields all agents—dominant and dominated—share an understanding of what is at stake in the field and tend to accept its principle of hierarchization. They are invested in the game, have an interest in it, and share the field’s illusio .

In the present case, we can interpret the various disciplines in the social sciences as more or less autonomous spaces that revolve around the shared stake in producing legitimate scientific knowledge by the standards of the field. What constitutes legitimate knowledge in these disciplinary fields, the production of which bestows scholars with prestige and an aura of competence, is in large part determined by the dominant agents in the field, who occupy positions in which most of the consecration of scientific work takes place. Scholars operating in a field are endowed with initial and accumulated field-specific capital, and are engaged in the struggle to gain additional capital (mainly scientific and intellectual prestige) in order to advance their position in the field. The main focus of these agents will generally be the disciplinary field in which they built their careers and invested their capital. These various disciplinary spaces are in turn part of a broader field of the social sciences in which the social status and prestige of the various disciplines is at stake. The ensuing disciplinary hierarchy is an important factor to take into account when analysing the circulation of new scientific products such as MMR. Furthermore, a distinction needs to be made between the academic and the scientific field. While the academic field revolves around universities and other degree-granting institutions, the stakes in the scientific field entail the production and valuation of knowledge. Of course, in modern science these fields are closely related, but they do not coincide (Gingras and Gemme 2006 ). For instance, part of the production of legitimate knowledge takes place outside of universities.

This framework makes it possible to contextualize the emergence of MMR in a socio-historical way. It also enables an assessment of some of the characteristics of MMR as a scientific product, since Bourdieu insists on the homology between the objective positions in a field and the position-takings of the agents who occupy these positions. As a new methodological approach, MMR is the result of the position-takings of its producers. The position-takings of the entrepreneurs at the core of MMR can therefore be seen as expressions in the struggles over the authority to define the proper methodology that underlies good scientific work regarding combining methods, and the potential rewards that come with being seen, by other agents, as authoritative on these matters. Possible rewards include a strengthened autonomy of the subfield of MMR and an improved position in the social-scientific field.

The role of these entrepreneurs or ‘intellectual leaders’ who can channel intellectual energy and can take the lead in institution building has been emphasised by sociologists of science as an important aspect of the production of knowledge that is visible and recognized as distinct in the larger scientific field (e.g., Mullins 1973 ; Collins 1998 ). According to Bourdieu, their position can, to a certain degree, explain the strategy they pursue and the options they perceive to be viable in the trade-off regarding the risks and potential rewards for their work.

We do not provide a full-fledged field analysis of MMR here. Rather, we use the concept as a heuristic device to account for the phenomenon of MMR in the social context in which it emerged and diffused. But first, we take stock of the current situation of MMR by focusing on the degree of institutionalization of MMR in the scientific field.

The institutionalization of mixed methods research

When discussing institutionalization, we have to be careful about what we mean by this term. More precisely, we need to be specific about the context and distinguish between institutionalization in the academic field and institutionalization within the scientific field (see Gingras and Gemme 2006 ; Sapiro et al. 2018 ). The first process refers to the establishment of degrees, curricula, faculties, etc., or to institutions tied to the academic bureaucracy and academic politics. The latter refers to the emergence of institutions that support the autonomization of scholarship such as scholarly associations and scientific journals. Since MMR is still a relatively young phenomenon and academic institutionalization tends to lag scientific institutionalization (e.g., for the case of sociology and psychology, see Sapiro et al. 2018 , p. 26), we mainly focus here on the latter dimension.

Drawing on criteria proposed by Fleck et al. ( 2016 ) for the institutionalization of academic disciplines, MMR seems to have achieved a significant degree of institutionalization within the scientific field. MMR quickly gained popularity in the first decade of the twenty-first century (e.g., Tashakkori and Teddlie 2010c , pp. 803–804). A distinct corpus of publications has been produced that aims to educate those interested in MMR and to function as a source of reference for researchers: there are a number of textbooks (e.g., Plowright 2010 ; Creswell and Plano Clark 2011 ; Teddlie and Tashakkori 2008 ); a handbook that is now in its second edition (Tashakkori and Teddlie 2003 , 2010a ); as well as a reader (Plano Clark and Creswell 2007 ). Furthermore, a journal (the Journal of Mixed Methods Research [ JMMR] ) was established in 2007. The JMMR was founded by the editors John Creswell and Abbas Tashakkori with the primary aim of “building an international and multidisciplinary community of mixed methods researchers.” Footnote 2 Contributions to the journal must “fit the definition of mixed methods research” Footnote 3 and explicitly integrate qualitative and quantitative aspects of research, either in an empirical study or in a more theoretical-methodologically oriented piece.

In addition, general textbooks on social research methods and methodology now increasingly devote sections to the issue of combining methods (e.g., Creswell 2008 ; Nagy Hesse-Biber and Leavy 2008 ; Bryman 2012 ), and MMR has been described as a “third paradigm” (Denscombe 2008 ), a “movement” (Bryman 2009 ), a “third methodology” (Tashakkori and Teddlie 2010b ), a “distinct approach” (Greene 2008 ) and an “emerging field” (Tashakkori and Teddlie 2011 ), defined by a common name (that sets it apart from other approaches to combining methods) and shared terminology (Tashakkori and Teddlie 2010b , p. 19). As a further indication of institutionalization, a research association (the Mixed Methods International Research Association—MMIRA) was founded in 2013 and its inaugural conference was held in 2014. Prior to this, there have been a number of conferences on MMR or occasions on which MMR was presented and discussed in other contexts. An example of the first is the conference on mixed method research design held in Basel in 2005. Starting also in 2005, the British Homerton School of Health Studies has organised a series of international conferences on mixed methods. Moreover, MMR was on the list of sessions in a number of conferences on qualitative research (see, e.g., Creswell 2012 ).

Another sign of institutionalization can be found in efforts to forge a common disciplinary identity by providing a narrative about its history. This involves the identification of precursors and pioneers as well as an interpretation of the process that gave rise to a distinctive set of ideas and practices. An explicit attempt to chart the early history of MMR is provided by Johnson and Gray ( 2010 ). They frame MMR as rooted in the philosophy of science, particularly as a way of thinking about science that has transcended some of the most salient historical oppositions in philosophy. Philosophers like Aristotle and Kant are portrayed as thinkers who sought to integrate opposing stances, forwarding “proto-mixed methods ideas” that exhibited the spirit of MMR (Johnson and Gray 2010 , p. 72, p. 86). In this capacity, they (as well as other philosophers like Vico and Montesquieu) are presented as part of MMR providing a philosophical validation of the project by presenting it as a continuation of ideas that have already been voiced by great thinkers in the past.

In the second edition of their textbook, Creswell and Plano Clark ( 2011 ) provide an overview of the history of MMR by identifying five historical stages: the first one being a precursor to the MMR approach, consisting of rather atomised attempts by different authors to combine methods in their research. For Creswell and Plano Clark, one of the earliest examples is Campbell and Fiske’s ( 1959 ) combination of quantitative methods to improve the validity of psychological scales that gave rise to the triangulation approach to research. However, they regard this and other studies that combined methods around that time, as “antecedents to (…) more systematic attempts to forge mixed methods into a complete research design” (Creswell and Plano Clark 2011 , p. 21), and hence label this stage as the “formative period” (ibid., p. 25). Their second stage consists of the emergence of MMR as an identifiable research strand, accompanied by a “paradigm debate” about the possibility of combining qualitative and quantitative data. They locate its beginnings in the late 1980s when researchers in various fields began to combine qualitative and quantitative methods (ibid., pp. 20–21). This provoked a discussion about the feasibility of combining data that were viewed as coming from very different philosophical points of view. The third stage, the “procedural development period,” saw an emphasis on developing more hands-on procedures for designing a mixed methods study, while stage four is identified as consisting of “advocacy and expansion” of MMR as a separate methodology, involving conferences, the establishment of a journal and the first edition of the aforementioned handbook (Tashakkori and Teddlie 2003 ). Finally, the fifth stage is seen as a “reflective period,” in which discussions about the unique philosophical underpinnings and the scientific position of MMR emerge.

Creswell and Plano Clark thus locate the emergence of “MMR proper” at the second stage, when researchers started to use both qualitative and quantitative methods within a single research effort. As reasons for the emergence of MMR at this stage they identify the growing complexity of research problems, the perception of qualitative research as a legitimate form of inquiry (also by quantitative researchers) and the increasing need qualitative researchers felt for generalising their findings. They therefore perceive the emergence of the practice of combining methods as a bottom up process that grew out of research practices, and at some point in time converged towards a more structural approach. Footnote 4 Historical accounts such as these add a cognitive dimension to the efforts to institutionalize MMR. They lay the groundwork for MMR as a separate subfield with its own identity, topics, problems and intellectual history. The use of terms such as “third paradigm” and “third methodology” also suggests that there is a tendency to perceive and promote MMR as a distinct and coherent way to do research.

In view of the brief exploration of the indicators of institutionalisation of MMR, it seems reasonable to conclude that MMR has become a recognizable and fairly institutionalized strand of research with its own identity and profile within the social scientific field. This can be seen both from the establishment of formal institutions (like associations and journals) and more informal ones that rely more on the tacit agreement between agents about “what MMR is” (an example of this, which we address later in the article, is the search for a common definition of MMR in order to fix the meaning of the term). The establishment of these institutions supports the autonomization of MMR and its emancipation from the field in which it originated, but in which it continues to be embedded. This way, it can be viewed as a semi-autonomous subfield within the larger field of the social sciences and as the result of a differentiation internal to this field (Steinmetz 2016 , p. 109). It is a space that is clearly embedded within this higher level field; for example, members of the subfield of MMR also qualify as members of the overarching field, and the allocation of the most valuable and current form of capital is determined there as well. Nevertheless, as a distinct subfield, it also has specific principles that govern the production of knowledge and the rewards of domination.

We return to the content and form of this specific knowledge later in the article. The next section addresses the question of the socio-genesis of MMR.

Where does mixed methods research come from?

The origins of the subfield of MMR lay in the broader field of social scientific disciplines. We interpret the positions of the scholars most involved in MMR (the “pioneers” or “scientific entrepreneurs”) as occupying particular positions within the larger academic and scientific field. Who, then, are the researchers at the heart of MMR? Leech ( 2010 ) interviewed 4 scholars (out of 6) that she identified as early developers of the field: Alan Bryman (UK; sociology), John Creswell (USA; educational psychology), Jennifer Greene (USA; educational psychology) and Janice Morse (USA; nursing and anthropology). Educated in the 1970s and early 1980s, all four of them indicated that they were initially trained in “quantitative methods” and later acquired skills in “qualitative methods.” For two of them (Bryman and Creswell) the impetus to learn qualitative methods was their involvement in writing on, and teaching of, research methods; for Greene and Morse the initial motivation was more instrumental and related to their concrete research activity at the time. Creswell describes himself as “a postpositivist in the 1970s, self-education as a constructivist through teaching qualitative courses in the 1980s, and advocacy for mixed methods (…) from the 1990s to the present” (Creswell 2011 , p. 269). Of this group, only Morse had the benefit of learning about qualitative methods as part of her educational training (in nursing and anthropology; Leech 2010 , p. 267). Independently, Creswell ( 2012 ) identified (in addition to Bryman, Greene and Morse) John Hunter, Allen Brewer (USA; Northwestern and Boston College) and Nigel Fielding (University of Surrey, UK) as important early movers in MMR.

The selections that Leech and Creswell make regarding the key actors are based on their close involvement with the “MMR movement.” It is corroborated by a simple analysis of the articles that appeared in the Journal of Mixed Methods Research ( JMMR ), founded in 2007 as an outlet for MMR.

Table 1 lists all the authors that have published in the issues of the journal since its first publication in 2007 and that have either received more than 14 (4%) of the citations allocated between the group of 343 authors (the TLCS score in Table 1 ), or have written more than 2 articles for the Journal (1.2% of all the articles that have appeared from 2007 until October 2013) together with their educational background (i.e., the discipline in which they completed their PhD).

All the members of Leech’s selection, except for Morse, and the members of Creswell’s selection (except Hunter, Brewer, and Fielding) are represented in the selection based on the entries in the JMMR . Footnote 5 The same holds for two of the three additional authors identified by Creswell. Hunter and Brewer have developed a somewhat different approach to combining methods that explicitly targets data gathering techniques and largely avoids epistemological discussions. In Brewer and Hunter ( 2006 ) they discuss the MMR approach very briefly and only include two references in their bibliography to the handbook of Tashakkori and Teddlie ( 2003 ), and at the end of 2013 they had not published in the JMMR . Fielding, meanwhile, has written two articles for the JMMR (Fielding and Cisneros-Puebla 2009 ; Fielding 2012 ). In general, it seems reasonable to assume that a publication in a journal that positions itself as part of a systematic attempt to build a research tradition, and can be viewed as part of a strategic effort to advance MMR as a distinct alternative to more “traditional” academic research—particularly in methods—at least signals a degree of adherence to the effort and acceptance of the rules of the game it lays out. This would locate Fielding closer to the MMR movement than the others.

The majority of the researchers listed in Table 1 have a background in psychology or social psychology (35%), and sociology (25%). Most of them work in the United States or are UK citizens, and the positions they occupied at the beginning of 2013 indicates that most of these are in applied research: educational research and educational psychology account for 50% of all the disciplinary occupations of the group that were still employed in academia. This is consistent with the view that MMR originated in applied disciplines and thematic studies like education and nursing, rather than “pure disciplines” like psychology and sociology (Tashakkori and Teddlie ( 2010b ), p. 32). Although most of the 20 individuals mentioned in Table 1 have taught methods courses in academic curricula (for 15 of them, we could determine that they were involved in the teaching of qualitative, quantitative, or mixed methods), there are few individuals with a background in statistics or a neighbouring discipline: only Amy Dellinger did her PhD in “research methodology.” In addition, as far as we could determine, only three individuals held a position in a methodological department at some time: Dellinger, Tony Onwuegbuzie, and Nancy Leech.

The pre-eminence of applied fields in MMR is supported when we turn our attention to the circulation of MMR. To assess this we proceeded as follows. We selected 10 categories in the Web of Science that form a rough representation of the space of social science disciplines, taking care to include the most important so-called “studies.” These thematically orientated, interdisciplinary research areas have progressively expanded since they emerged at the end of the 1960s as a critique of the traditional disciplines (Heilbron et al. 2017 ). For each category, we selected the 10 journals with the highest 5-year impact factor in their category in the period 2007–2015. The lists were compiled bi-annually over this period, resulting in 5 top ten lists for the following Web of Science categories: Economics, Psychology, Sociology, Anthropology, Political Science, Nursing, Education & Educational Research, Business, Cultural Studies, and Family Studies. After removing multiple occurring journals, we obtained a list of 164 journals.

We searched the titles and abstracts of the articles appearing in these journals over the period 1992–2016 for occurrences of the terms “mixed method” or “multiple methods” and variants thereof. We chose this particular period and combination of search terms to see if a shift from a more general use of the term “multiple methods” to “mixed methods” occurred following the institutionalization of MMR. In total, we found 797 articles (out of a total of 241,521 articles that appeared in these journals during that time), published in 95 different journals. Table 2 lists the 20 journals that contain at least 1% (8 articles) of the total amount of articles.

As is clear from Table 2 , the largest number of articles in the sample were published in journals in the field of nursing: 332 articles (42%) appeared in journals that can be assigned to this category. The next largest category is Education & Educational Research, to which 224 (28 percentage) of the articles can be allocated. By contrast, classical social science disciples are barely represented. In Table 2 only the journal Field Methods (Anthropology) and the Journal of Child Psychology and Psychiatry (Psychology) are related to classical disciplines. In Table 3 , the articles in the sample are categorized according to the disciplinary category of the journal in which they appeared. Overall, the traditional disciplines are clearly underrepresented: for the Economics category, for example, only the Journal of Economic Geography contains three articles that make a reference to mixed methods.

Focusing on the core MMR group, the top ten authors of the group together collect 458 citations from the 797 articles in the sample, locating them at the center of the citation network. Creswell is the most cited author (210 citations) and his work too receives most citations from journals in nursing and education studies.

The question whether a terminological shift has occurred from “multiple methods” to “mixed methods” must be answered affirmative for this sample. Prior to 2001 most articles (23 out of 31) refer to “multiple methods” or “multi-method” in their title or abstract, while the term “mixed methods” gains traction after 2001. This shift occurs first in journals in nursing studies, with journals in education studies following somewhat later. The same fields are also the first to cite the first textbooks and handbooks of MMR.

Taken together, these results corroborate the notion that MMR circulates mainly in nursing and education studies. How can this be understood from a field theoretical perspective? MMR can be seen as an innovation in the social scientific field, introducing a new methodology for combining existing methods in research. In general, innovation is a relatively risky strategy. Coming up with a truly rule-breaking innovation often involves a small probability of great success and a large probability of failure. However, it is important to add some nuance to this general observation. First, the risk an innovator faces depends on her position in the field. Agents occupying positions at the top of their field’s hierarchy are rich in specific capital and can more easily afford to undertake risky projects. In the scientific field, these are the agents richest in scientific capital. They have the knowledge, authority, and reputation (derived from recognition by their peers; Bourdieu 2004 , p. 34) that tends to decrease the risk they face and increase the chances of success. Moreover, the positions richest in scientific capital will, by definition, be the most consecrated ones. This consecration involves scientific rather than academic capital (cf. Wacquant 2013 , p. 20) and within disciplines these consecrated positions often are related to orthodox position-takings. This presents a paradox: although they have the capital to take more risks, they have also invested heavily in the orthodoxy of the field and will thus be reluctant to upset the status quo and risk destroying the value of their investment. This results in a tendency to take a more conservative stance, aimed at preserving the status quo in the field and defending their position. Footnote 6

For agents in dominated positions this logic is reversed. Possessing less scientific capital, they hold less consecrated positions and their chances of introducing successful innovations are much lower. This leaves them too with two possible strategies. One is to revert to a strategy of adaptation, accepting the established hierarchy in the field and embarking on a slow advancement to gain the necessary capital to make their mark from within the established order. However, Bourdieu notes that sometimes agents with a relatively marginal position in the field will engage in a “flight forward” and pursue higher risk strategies. Strategies promoting a heterodox approach challenge the orthodoxy and the principles of hierarchization of the field, and, if successful (which will be the case only with a small probability), can rake in significant profits by laying claim to a new orthodoxy (Bourdieu 1975 , p. 104; Bourdieu 1993 , pp. 116–117).

Thus, the coupling of innovative strategies to specific field positions based on the amount of scientific capital alone is not straightforward. It is therefore helpful to introduce a second differentiation in the field that, following Bourdieu ( 1975 , p. 103), is based on the differences between the expected profits from these strategies. Here a distinction can be made between an autonomous and a heteronomous pole of the field, i.e., between the purest, most “disinterested” positions and the most “temporal” positions that are more pervious to the heteronomous logic of social hierarchies outside the scientific field. Of course, this difference is a matter of degree, as even the works produced at the most heteronomous positions still have to adhere to the standards of the scientific field to be seen as legitimate. But within each discipline this dimension captures the difference between agents predominantly engaged in fundamental, scholarly work—“production solely for the producers”—and agents more involved in applied lines of research. The main component of the expected profit from innovation in the first case is scientific, whereas in the second case the balance tends to shift towards more temporal profits. This two-fold structuring of the field allows for a more nuanced conception of innovation than the dichotomy “conservative” versus “radical.” Holders of large amounts of scientific capital at the autonomous pole of the field are the producers and conservators of orthodoxy, producing and diffusing what can be called “orthodox innovations” through their control of relatively powerful networks of consecration and circulation. Innovations can be radical or revolutionary in a rational sense, but they tend to originate from questions raised by the orthodoxy of the field. Likewise, the strategy to innovate in this sense can be very risky in that success is in no way guaranteed, but the risk is mitigated by the assurance of peers that these are legitimate questions, tackled in a way that is consistent with orthodoxy and that does not threaten control of the consecration and circulation networks.

These producers are seen as intellectual leaders by most agents in the field, especially by those aspiring to become part of the specific networks of production and circulation they maintain. The exception are the agents located at the autonomous end of the field who possess less scientific capital and outright reject this orthodoxy produced by the field’s elite. Being strictly focused on the most autonomous principles of legitimacy, they are unable to accommodate and have no choice but to reject the orthodoxy. Their only hope is to engage in heterodox innovations that may one day become the new orthodoxy.

The issue is less antagonistic at the heteronomous side of the field, at least as far as the irreconcilable position-takings at the autonomous pole are concerned. The main battle here is also for scientific capital, but is complemented by the legitimacy it brings to gain access to those who are in power outside of the scientific field. At the dominant side, those with more scientific capital tend to have access to the field of power, agents who hold the most economic and cultural capital, for example by holding positions in policy advisory committees or company boards. The dominated groups at this side of the field will cater more to practitioners or professionals outside of the field of science.

Overall, there will be fewer innovations on this side. Moreover, innovative strategies will be less concerned with the intricacies of the pure discussions that prevail at the autonomous pole and be of a more practical nature, but pursued from different degrees of legitimacy according to the differences in scientific capital. This affects the form these more practical, process-orientated innovations take. At the dominant side of this pole, agents tend to accept the outcome of the struggles at the autonomous pole: they will accept the orthodoxy because mastery of this provides them with scientific capital and the legitimacy they need to gain access to those in power. In contrast, agents at the dominated side will be more interested in doing “what works,” neutralizing the points of conflict at the autonomous pole and deriving less value from strictly following the orthodoxy. This way, a four-fold classification of innovative strategies in the scientific field emerges (see Fig.  2 ) that helps to understand the context in which MMR was developed.

Scientific field and scientific innovation

In summary, the small group of researchers who have been identified as the core of MMR consist predominantly of users of methods, who were educated and have worked exclusively at US and British universities. The specific approach to combining methods that is proposed by MMR has been successful from an institutional point of view, achieving visibility through the foundation of a journal and association and a considerable output of core MMR scholars in terms of books, conference proceedings, and journal articles. Its origins and circulation in vocational studies rather than classical academic disciplines can be understood from the position these studies occupy in the scientific field and the kinds of position-taking and innovations these positions give rise to. This context allows a reflexive understanding of the content of MMR and the issues that are dominant in the approach. We turn to this in the next section.

Mixed methods research: Position-taking

The position of the subfield of MMR in the scientific field is related to the position-takings of agents that form the core of this subfield (Bourdieu 1993 , p. 35). The space of position takings, in turn, provides the framework to study the most salient issues that are debated within the subfield. Since we can consider MMR to be an emerging subfield, where positions and position takings are not as clearly defined as in more mature and settled fields, it comes as no surprise that there is a lively discussion of fundamental matters. Out of the various topics that are actively discussed, we have distilled three themes that are important for the way the subfield of MMR conveys its autonomy as a field and as a distinct approach to research. Footnote 7 In our view, these also represent the main problems with the way MMR approaches the issue of combining methods.

Methodology making and standardization

The first topic is that the approach is moving towards defining a unified MMR methodology. There are differences in opinion as to how this is best achieved, but there is widespread agreement that some kind of common methodological and conceptual foundation of MMR is needed. To this end, some propose a broad methodology that can serve as distinct marker of MMR research. For instance, in their introduction to the handbook, Tashakkori and Teddlie ( 2010b ) propose a definition of the methodology of mixed methods research as “the broad inquiry logic that guides the selection of specific methods and that is informed by conceptual positions common to mixed methods practitioners” (Tashakkori and Teddlie 2010b , p. 5). When they (later on in the text) provide two methodological principles that differentiate MMR from other communities of scholars, they state that they regard it as a “crucial mission” for the MMR community to generate distinct methodological principles (Tashakkori and Teddlie 2010b , pp. 16–17). They envision an MMR methodology that can function as a “guide” for selecting specific methods. Others are more in favour of finding a philosophical foundation that underlies MMR. For instance, Morgan ( 2007 ) and Hesse-Biber ( 2010 ) consider pragmatism as a philosophy that distinguishes MMR from qualitative (constructivism) and quantitative (positivist) research and that can provide a rationale for the paradigmatic pluralism typical of MMR.

Furthermore, there is wide agreement that some unified definition of MMR would be beneficial, but it is precisely here that there is a large variation in interpretations regarding the essentials of MMR. This can be seen in the plethora of definitions that have been proposed. Johnson et al. ( 2007 ) identified 19 alternative definitions of MMR at the time, out of which they condensed their own:

[MMR] is the type of research in which a researcher or team of researchers combines elements of qualitative and quantitative research approaches (e.g., use of qualitative and quantitative viewpoints, data collection, analysis, inference techniques) for the broad purpose of breath and depth of understanding and corroboration. Footnote 8

Four years later, the issue is not settled yet. Creswell and Plano Clark ( 2011 ) list a number of authors who have proposed a different definition of MMR, and conclude that there is a common trend in the content of these definitions over time. They take the view that earlier texts on mixing methods stressed a “disentanglement of methods and philosophy,” while later texts locate the practice of mixing methods in “all phases of the research process” (Creswell and Plano Clark 2011 , p. 2). It would seem, then, that according to these authors the definitions of MMR have become more abstract, further away from the practicality of “merely” combining methods. Specifically, researchers now seem to speak of mixing higher order concepts: some speak of mixing methodologies, others refer to mixing “research approaches,” or combining “types of research,” or engage in “multiple ways of seeing the social world” (Creswell and Plano Clark 2011 ).

This shift is in line with the direction in which MMR has developed and that emphasises practical ‘manuals’ and schemas for conducting research. A relatively large portion of the MMR literature is devoted to classifications of mixed methods designs. These classifications provide the basis for typologies that, in turn, provide guidelines to conduct MMR in a concrete research project. Tashakkori and Teddlie ( 2003 ) view these typologies as important elements of the organizational structure and legitimacy of the field. In addition, Leech and Onwuegbuzie ( 2009 ) see typologies as helpful guides for researchers and of pedagogical value (Leech and Onwuegbuzie 2009 , p. 272). Proposals for typologies can be found in textbooks, articles, and contributions to the handbook(s). For example, Creswell et al. ( 2003 , pp. 169-170) reviewed a number of studies and identified 8 different ways to classify MMR studies. This list was updated and extended by Creswell and Plano Clark ( 2011 , pp. 56-59) to 15 typologies. Leech and Onwuegbuzie ( 2009 ) identified 35 different research designs in the contributions to Teddlie and Tashakkori (2003) alone, and proposed their own three-dimensional typology that resulted in 8 different types of mixed methods studies. As another example of the ubiquity of these typologies, Nastasi et al. ( 2010 ) classified a large number of existing typologies in MMR into 7”meta-typologies” that each emphasize different aspects of the research process as important markers for MMR. According to the authors, these typologies have the same function in MMR as the more familiar names of “qualitative” or “quantitative” methods (e.g., “content analysis” or “structural equation modelling”) have: to signal readers of research what is going on, what procedures have been followed, how to interpret results, etc. (see also Creswell et al. 2003 , pp. 162–163). The criteria underlying these typologies mainly have to do with the degree of mixing (e.g., are methods mixed throughout the research project or not?), the timing (e.g., sequential or concurrent mixing of methods) and the emphasis (e.g., is one approach dominant, or do they have equal status?).

We find this strong drive to develop methodologies, definitions, and typologies of MMR as guides to valid mixed methods research problematic. What it amounts to in practice is a methodology that lays out the basic guidelines for doing MMR in a “proper way.” This entails the danger of straight-jacketing reflection about the use of methods, decoupling it from theoretical and empirical considerations, thus favouring the unreflexive use of a standard methodology. Researchers are asked to make a choice for a particular MMR design and adhere to the guidelines for a “proper” MMR study. Such methodological prescription diametrically opposes the initial critique of the mechanical and unreflexive use of methods. The insight offered by Bourdieu’s notion of reflexivity is, on the contrary, that the actual research practice is fundamentally open in terms of being guided by a logic of practice that cannot be captured by a preconceived and all-encompassing logic independent of that practice. Reflexivity in this view cannot be achieved by hiding behind the construct of a standardized methodology—of whatever signature—it can only be achieved by objectifying the process of objectification that goes on within the context of the field in which the researcher is embedded. This reflexivity, then, requires an analysis of the position of the researcher as a critical component of the research process, both as the embodiment of past choices that have consequences for the strategic position in the scientific field, and as predispositions regarding the choice for the subject and content of a research project. By adding the insight of STS researchers that the point of deconstructing science and technology is not so much to offer a new best way of doing science or technology, but to provide insights into the critical moments in research (for a take on such a debate, see, for example, Edge 1995 , pp. 16–20), this calls for a sociology of science that takes methods much more seriously as objects of study. Such a programme should be based on studying the process of codification and standardization of methods in their historical context of production, circulation, and use. It would provide a basis for a sociological understanding of methods that can illuminate the critical moments in research alluded to above, enabling a systematic reflection on the process of objectification. This, in turn, allows a more sophisticated validation of using—and combining—methods than relying on prescribed methodologies.

The role of epistemology

The second theme discussed in a large number of contributions is the role epistemology plays in MMR. In a sense, epistemology provides the lifeblood for MMR in that methods in MMR are mainly seen in epistemological terms. This interpretation of methods is at the core of the knowledge claim of MMR practitioners, i.e., that the mixing of methods means mixing broad, different ways of knowing, which leads to better knowledge of the research object. It is also part of the identity that MMR consciously assumes, and that serves to set it apart from previous, more practical attempts to combine methods. This can be seen in the historical overview that Creswell and Plano Clark ( 2011 ) presented and that was discussed above. This reading, in which combining methods has evolved from the rather unproblematic level (one could alternatively say “naïve” or “unaware”) of instrumental use of various tools and techniques into an act that requires deeper thinking on a methodological and epistemological level, provides the legitimacy of MMR.

At the core of the MMR approach we thus find that methods are seen as unproblematic representations of different epistemologies. But this leads to a paradox, since the epistemological frameworks need to be held flexible enough to allow researchers to integrate elements of each of them (in the shape of methods) into one MMR design. As a consequence, the issue becomes the following: methods need to be disengaged from too strict an interpretation of the epistemological context in which they were developed in order for them to be “mixable,”’, but, at the same time, they must keep the epistemology attributed to them firmly intact.

In the MMR discourse two epistemological positions are identified that matter most: a positivist approach that gives rise to quantitative methods and a constructivist approach that is home to qualitative methods. For MMR to be a feasible endeavour, the differences between both forms of research must be defined as reconcilable. This position necessitates an engagement with those who hold that the quantitative/qualitative dichotomy is unbridgeable. Within MMR an interesting way of doing so has emerged. In the first issue of the Journal of Mixed Methods Research, Morgan ( 2007 ) frames the debate about research methodology in the social sciences in terms of Kuhnian paradigms, and he argues that the pioneers of the emancipation of qualitative research methods used a particular interpretation of the paradigm-concept to state their case against the then dominant paradigm in the social sciences. According to Morgan, they interpreted a paradigm mainly in metaphysical terms, stressing the connections among the trinity of ontology, epistemology, and methodology as used in the philosophy of knowledge (Morgan 2007 , p. 57). This allowed these scholars to depict the line between research traditions in stark, contrasting terms, using Kuhn’s idea of “incommensurability” in the sense of its “early Kuhn” interpretation. This strategy fixed the contrast between the proposed alternative approach (a “constructivist paradigm”), and the traditional approach (constructed as “the positivist paradigm”) to research as a whole, and offered the alternative approach as a valid option rooted in the philosophy of knowledge. Morgan focuses especially on the work of Egon Guba and Yvonne Lincoln who developed what they initially termed a “naturalistic paradigm” as an alternative to their perception of positivism in the social sciences (e.g., Guba and Lincoln 1985 ). Footnote 9 MMR requires a more flexible or “a-paradigmatic stance” towards research, which would entail that “in real-world practice, methods can be separated from the epistemology out of which they emerged” (Patton 2002 , quoted in Tashakkori and Teddlie 2010b , p. 14).

This proposal of an ‘interpretative flexibility’ (Bijker 1987 , 1997 ) regarding paradigms is an interesting proposition. But it immediately raises the question: why stop there? Why not take a deeper look into the epistemological technology of methods themselves, to let the muted components speak up in order to look for alternative “mixing interfaces” that could potentially provide equally valid benefits in terms of the understanding of a research object? The answer, of course, was already seen above. It is that the MMR approach requires situating methods epistemologically in order to keep them intact as unproblematic mediators of specific epistemologies and, thus, make the methodological prescriptions work. There are several problems with this. First, seeing methods solely through an epistemological lens is problematic, but it would be less consequential if it were applied to multiple elements of methods separately. This would at least allow a look under the hood of a method, and new ways of mixing methods could be opened up that go beyond the crude “qualitative” versus “quantitative” dichotomy. Second, there is also the issue of the ontological dimension of methods that is disregarded in an exclusively epistemological framing of methods (e.g., Law 2004 ). Taking this ontological dimension seriously has at least two important facets. First, it draws attention to the ontological assumptions that are woven into methods in their respective fields of production and that are imported into fields of users. Second, it entails the ontological consequences of practising methods: using, applying, and referring to methods and the realities this produces. This latter facet brings the world-making and boundary-drawing capacities of methods to the fore. Both facets are ignored in MMR. We say more about the first facet in the next section. With regard to the second facet, a crucial element concerns the data that are generated, collected, and analysed in a research project. But rather than problematizing the link between the performativity of methods and the data that are enacted within the frame of a method, here too MMR relies on a dichotomy: that between quantitative and qualitative data. Methods are primarily viewed as ways of gathering data or as analytic techniques dealing with a specific kind of data. Methods and data are conceptualised intertwiningly: methods too are seen as either quantitative or qualitative (often written as QUANT and QUAL in the literature), and perform the role of linking epistemology and data. In the final analysis, the MMR approach is based on the epistemological legitimization of the dichotomy between qualitative and quantitative data in order to define and combine methods: data obtain epistemological currency through the supposed in-severable link to certain methods, and methods are reduced to the role of acting as neutral mediators between them.

In this way, methods are effectively reduced to, on the one hand, placeholders for epistemological paradigms and, on the other hand, mediators between one kind of data and the appropriate epistemology. To put it bluntly, the name “mixed methods research” is actually a misnomer, because what is mixed are paradigms or “approaches,” not methods. Thus, the act of mixing methods à la MMR has the paradoxical effect of encouraging a crude black box approach to methods. This is a third problematic characteristic of MMR, because it hinders a detailed study of methods that can lead to a much richer perspective on mixing methods.

Black boxed methods and how to open them

The third problem that we identified with the MMR approach, then, is that with the impetus to standardize the MMR methodology by fixing methods epistemologically, complemented by a dichotomous view of data, they are, in the words of philosopher Bruno Latour, “blackboxed.” This is a peculiar result of the prescription for mixing methods as proposed by MMR that thus not only denies practice and the ontological dimensions of methods and data, but also casts methods in the role of unyielding black boxes. Footnote 10 With this in mind, it will come as no surprise that most foundational contributions to the MMR literature do not explicitly define what a method is, nor that they do not provide an elaborative historical account of individual methods. The particular framing of methods in MMR results in a blind spot for the historical and social context of the production and circulation of methods as intellectual products. Instead it chooses to reify the boundaries that are drawn between “qualitative” and “quantitative” methods and reproduce them in the methodology it proposes. Footnote 11 This is an example of “circulation without context” (Bourdieu 2002 , p. 4): classifications that are constructed in the field of use or reception without taking the constellation within the field of production seriously.

Of course, this does not mean that the reality of the differences between quantitative and qualitative research must be denied. These labels are sticky and symbolically laden. They have come, in many ways, to represent “two cultures” (Goertz and Mahony 2012 ) of research, institutionalised in academia, and the effects of nominally “belonging” to (or being assigned to) one particular category have very real consequences in terms of, for instance, access to research grants and specific journals. However, if the goal of an approach such as MMR is to open up new pathways in social science research, (and why should that not be the case?) it is hard to see how that is accomplished by defining the act of combining methods solely in terms of reified differences between research using qualitative and quantitative data. In our view, methods are far richer and more interesting constructs than that, and a practice of combining methods in research should reflect that. Footnote 12

Addressing these problems entices a reflection on methods and using (multiple) methods that is missing in the MMR perspective. A fruitful way to open up the black boxes and take into account the epistemological and ontological facets of methods is to make them, and their use, the object of sociological-historical investigation. Methods are constituted through particular practices. In Bourdieusian terms, they are objectifications of the subjectively understood practices of scientists “in other fields.” Rather than basing a practice of combining methods on an uncritical acceptance of the historically grown classification of types of social research (and using these as the building stones of a methodology of mixing methods), we propose the development of a multifaceted approach that is based on a study of the different socio-historical contexts and practices in which methods developed and circulated.

A sociological understanding of methods based on these premises provides the tools to break with the dichotomously designed interface for combining methods in MMR. Instead, focusing on the historical and social contexts of production and use can reveal the traces that these contexts leave, both in the internal structure of methods, how they are perceived, how they are put into practice, and how this practice informs the ontological effects of methods. Seeing methods as complex technologies, with a history that entails the struggles among the different agents involved in their production, and use opens the way to identify multiple interfaces for combining them: the one-sided boxes become polyhedra. The critical study of methods as “objects of objectification” also entices analyses of the way in which methods intervene between subject (researcher) and object and the way in which different methods are employed in practice to draw this boundary differently. The reflexive position generated by such a systematic juxtaposition of methods is a fruitful basis to come to a richer perspective on combining methods.

We critically reviewed the emerging practice of combining methods under the label of MMR. MMR challenges the mono-method approaches that are still dominant in the social sciences, and this is both refreshing and important. Combining methods should indeed be taken much more seriously in the social sciences.

However, the direction that the practice of combining methods is taking under the MMR approach seems problematic to us. We identified three main concerns. First, MMR scholars seem to be committed to designing a standardized methodological framework for combining methods. This is unfortunate, since it amounts to enforcing an unnecessary codification of aspects of research practices that should not be formally standardized. Second, MMR constructs methods as unproblematic representations of an epistemology. Although methods must be separable from their native epistemology for MMR to work, at the same time they have to be nested within a qualitative or a quantitative research approach, which are characterized by the data they use. By this logic, combining quantitative methods with other quantitative methods, or qualitative methods with other qualitative methods, cannot offer the same benefits: they originate from the same way of viewing and knowing the world, so it would have the same effect as blending two gradations of the same colour paint. The importance attached to the epistemological grounding of methods and data in MMR also disregards the ontological aspects of methods. In this article, we are arguing that this one-sided perspective is problematic. Seeing combining methods as equivalent to combining epistemologies that are somehow pure and internally homogeneous because they can be placed in a qualitative or quantitative framework essentially amounts to reifying these categories.

It also leads to the third problem: the black boxing of methods as neutral mediators between these epistemologies and data. This not only constitutes a problem for trying to understand methods as intellectual products, but also for regarding the practice of combining methods, because it ignores the social-historical context of the development of individual methods and hinders a sociologically grounded notion of combining methods.

We proceed from a different perspective on methods. In our view, methods are complex constructions. They are world-making technologies that encapsulate different assumptions on causality, rely on different conceptual relations and categorizations, allow for different degrees of emergence, and employ different theories of the data that they internalise as objects of analysis. Even more importantly, their current form as intellectual products cannot be separated from the historical context of their production, circulation, and use.

A fully developed exposition of such an approach will have to await further work. Footnote 13 So far, the sociological study of methods has not (yet) developed into a consistent research programme, but important elements can be derived from existing contributions such as MacKenzie ( 1981 ), Chapoulie ( 1984 ), Platt ( 1996 ), Freeman ( 2004 ), and Desrosières ( 2008a , b ). The work on the “social life of methods” (e.g., Savage 2013 ) also contains important leads for the development of a systematic sociological approach to method production and circulation. Based on the discussion in this article and the contributions listed above, some tantalizing questions can be formulated. How are methods and their elements objectified? How are epistemology and ontology defined in different fields and how do those definitions feed into methods? How do they circulate and how are they translated and used in different contexts? What are the main controversies in fields of users and how are these related to the field of production? What are the homologies between these fields?

Setting out to answer these questions opens up the possibility of exploring other interesting combinations of methods that emerge from the combination of different practices, situated in different historical and epistemological contexts, and with their unique set of interpretations regarding their constituent elements. One of these must surely be the data-theoretical elements that different methods incorporate. The problematization of data has become all the more pressing now that the debate about the consequences of “big data” for social scientific practices has become prominent (Savage and Burrows 2007 ; Levallois et al. 2013 ; Burrows and Savage 2014 ). Whereas MMR emphasizes the dichotomy between qualitative and quantitative data, a historical analysis of the production and use of methods can explore the more subtle, different interpretations and enactments of the “same” data. These differences inform method construction, controversies surrounding methods and, hence, opportunities for combining methods. These could then be constructed based on alternative conceptualisations of data. Again, while in some contexts it might be enlightening to rely on the distinction between data as qualitative or quantitative, and to combine methods based on this categorization, it is an exciting possibility that in other research contexts other conceptualisations of data might be of more value to enhance a specific (contextual) form of knowledge.

Change history

06 may 2019.

Unfortunately, figure 2 was incorrectly published.

The search term used was “mixed method*” in the “topic” search field of SSCI, A&HCI, and CPCI-SSH as contained in the Web of Science. A Google NGram search (not shown) confirmed this pattern. The results of a search for “mixed methods” and “mixed methods research” showed a very steep increase after 1994: in the first case, the normalized share in the total corpus increased by 855% from 1994 till 2008. Also, Creswell ( 2012 ) reports an almost hundred-fold increase in the number of theses and dissertations with mixed methods’ in the citation and abstract (from 26 in 1990–1994 to 2524 in 2005–2009).

Retrieved from https://uk.sagepub.com/en-gb/eur/journal-of-mixed-methods-research/journal201775#aims-and-scope on 1/17/2019.

In terms of antecedents of mixed methods research, it is interesting to note that Bourdieu, whose sociology of science we draw on, was, from his earliest studies in Algeria onwards, a strong advocate of combining research methods. He made it into a central characteristic of his approach to social science in Bourdieu et al. ( 1991 [1968]). His approach, as we see below, was very different from the one now proposed under the banner of MMR. Significantly, there is no mention of Bourdieu’s take on combining methods in any of the sources we studied.

Morse’s example in particular warns us that restricting the analysis to the authors that have published in the JMMR runs the risk of missing some important contributors to the spread of MMR through the social sciences. On her website, Morse lists 11 publications (journal articles, book chapters, and books) that explicitly make reference to mixed methods (and a substantial number of other publications are about methodological aspects of research), so the fact that she has not (yet) published in the JMMR cannot, by itself, be taken as an indication of a lesser involvement with the practice of combining methods. See the website of Janice Morse at https://faculty.utah.edu/u0556920-Janice_Morse_RN,_PhD,_FAAN/hm/index.hml accessed 1/17/2019.

Bourdieu ( 1999 , p. 26) mentions that one has to be a scientific capitalist to be able to start a scientific revolution. But here he refers explicitly to the autonomy of the scientific field, making it virtually impossible for amateurs to stand up against the historically accumulated capital in the field and incite a revolution.

The themes summarize the key issues through which MMR as a group comes “into difference” (Bourdieu 1993 , p. 32). Of course, as in any (sub)field, the agents identified above often differ in their opinions on some of these key issues or disagree on the answer to the question if there should be a high degree of convergence of opinions at all. For instance, Bryman ( 2009 ) worried that MMR could become “a ghetto.” For him, the institutional landmarks of having a journal, conferences, and a handbook increase the risk of “not considering the whole range of possibilities.” He added: “I don’t regard it as a field, I kind of think of it as a way of thinking about how you go about research.” (Bryman, cited in Leech 2010 , p. 261). It is interesting to note that Bryman, like fellow sociologists Morgan and Denscombe, had published only one paper in the JMMR by the end of 2016 (Bryman passed away in June of 2017). Although these papers are among the most cited papers in the journal (see Table 1 ), this low number is consistent with the more eclectic approach that Bryman proposed.

Johnson, Onwuegbuzie, and Turner ( 2007 , p. 123).

Guba and Lincoln ( 1985 ) discuss the features of their version of a positivistic approach mainly in ontological and epistemological terms, but they are also careful to distinguish the opposition between naturalistic and positivist approaches from the difference between what they call the quantitative and the qualitative paradigms. Since they go on to state that, in principle, quantitative methods can be used within a naturalistic approach (although in practice, qualitative methods would be preferred by researchers embracing this paradigm), they seem to locate methods on a somewhat “lower,” i.e., less incommensurable level. However, in their later work (both together as well as with others or individually) and that of others in their wake, there seems to have been a shift towards a stricter interpretation of the qualitative/quantitative divide in metaphysical terms, enabling Teddlie and Tashakkori (2010b) to label this group “purists” (Tashakkori and Teddlie 2010b , p. 13).

See, for instance, Onwuegbuzie et al.’s ( 2011 ) classification of 58 qualitative data analysis techniques and 18 quantitative data analysis techniques.

This can also be seen in Morgan’s ( 2018 ) response to Sandelowski’s ( 2014 ) critique of the binary distinctions in MMR between qualitative and quantitative research approaches and methods. Morgan denounces the essentialist approach to categorizing qualitative and quantitative research in favor of a categorization based on “family resemblances,” in which he draws on Wittgenstein. However, this denies the fact that the essentialist way of categorizing is very common in the MMR corpus, particularly in textbooks and manuals (e.g., Plano Clark and Ivankova 2016 ). Moreover, and more importantly, he still does not extend this non-essentialist model of categorization to the level of methods, referring, for instance, to the different strengths of qualitative and quantitative methods in mixed methods studies (Morgan 2018 , p. 276).

While it goes beyond the scope of this article to delve into the history of the qualitative-quantitative divide in the social sciences, some broad observations can be made here. The history of method use in the social sciences can briefly be summarized as first, a rather fluid use of what can retrospectively be called different methods in large scale research projects—such as the Yankee City study of Lloyd Warner and his associates (see Platt 1996 , p. 102), the study on union democracy of Lipset et al. ( 1956 ), and the Marienthal study by Lazarsfeld and his associates (Jahoda et al. 1933 ); see Brewer and Hunter ( 2006 , p. xvi)—followed by an increasing emphasis on quantitative data and the objectification and standardization of methods. The rise of research using qualitative data can be understood as a reaction against this use and interpretation of method in the social sciences. However, out of the ensuing clash a new, still dominant classification of methods emerged, one that relies on the framing of methods as either “qualitative” or “quantitative.” Moreover, these labels have become synonymous with epistemological positions that are reproduced in MMR.

A proposal to come to such an approach can be found in Timans ( 2015 ).

Bijker, W. (1987). The social construction of bakelite: Toward a theory of invention. In W. Bijker, T. Hughes, T. Pinch, & D. Douglas (Eds.), The social construction of technological systems: New directions in the sociology and history of technology . Cambridge, MA: MIT press.

Google Scholar  

Bijker, W. (1997). Of bicycles, bakelites, and bulbs: Toward a theory of sociotechnical change . Cambridge, MA: MIT Press.

Bourdieu, P. (1975). La spécifité du champ scientifique et les conditions sociales du progrès de la raison. Sociologie et Sociétés, 7 (1), 91–118.

Article   Google Scholar  

Bourdieu, P. (1988). Homo academicus . Stanford, CA: Stanford University Press.

Bourdieu, P. (1993). The field of cultural production . Cambridge, UK: Polity Press.

Bourdieu, P. (1999). Les historiens et la sociologie de Pierre Bourdieu. Le Bulletin de la Société d'Histoire Moderne et Contemporaine/SHMC, 1999 (3&4), 4–27.

Bourdieu, P. (2002). Les conditions sociales de la circulation internationale des idées. Actes de la Recherche en Sciences Sociales, 145 (5), 3–8.

Bourdieu, P. (2004). Science of science and reflexivity . Cambridge, UK: Polity.

Bourdieu, P. (2007). Sketch for a self-analysis . Cambridge, UK: Polity.

Bourdieu, P., Chamboredon, J., & Passeron, J. (1991). The craft of sociology: Epistemological preliminaries . Berlin, Germany: De Gruyter.

Book   Google Scholar  

Brewer, J., & Hunter, A. (2006). Multimethod research: A synthesis of styles . London, UK: Sage.

Bryman, A. (2009). Sage Methodspace: Alan Bryman on research methods. Retrieved from http://www.youtube.com/watch?v=bHzM9RlO6j0 . Accessed 3/7/2019.

Bryman, A. (2012). Social research methods . Oxford, UK: Oxford University Press.

Burrows, R., & Savage, M. (2014). After the crisis? Big data and the methodological challenges of empirical sociology. Big Data & Society, 1 (1), 1–6.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56 (2), 81–105.

Chapoulie, J. (1984). Everett C. Hughes et le développement du travail de terrain en sociologie. Revue Française de Sociologie, 25 (4), 582–608.

Collins, R. (1998). The sociology of philosophies: A global theory of intellectual change . Cambridge, MA: Harvard University Press.

Creswell, J. (2008). Research design: Qualitative, quantitative, and mixed methods approaches . Thousand Oaks, CA: Sage.

Creswell, J. (2011). Controversies in mixed methods research. In N. Denzin & Y. Lincoln (Eds.), The Sage handbook of qualitative research . Thousand Oaks, CA: Sage.

Creswell, J. (2012). Qualitative inquiry and research design: Choosing among five approaches . Thousand Oaks, CA: Sage.

Creswell, J., & Plano Clark, V. (2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell, J., Plano Clark, V., Gutmann, M., & Hanson, W. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Denscombe, M. (2008). Communities of practice a research paradigm for the mixed methods approach. Journal of Mixed Methods Research, 2 (3), 270–283.

Desrosières, A. (2008a). Pour une sociologie historique de la quantification - L’Argument statistique I . Paris, France: Presses des Mines.

Desrosières, A. (2008b). Gouverner par les nombres - L’Argument statistique II . Paris, France: Presses des Mines.

Edge, D. (1995). Reinventing the wheel. In D. Edge, S. Jasanof, G. Markle, J. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies . Thousand Oaks, CA: Sage.

Fielding, N. (2012). Triangulation and mixed methods designs data integration with new research technologies. Journal of Mixed Methods Research, 6 (2), 124–136.

Fielding, N., & Cisneros-Puebla, C. (2009). CAQDAS-GIS convergence: Toward a new integrated mixed method research practice? Journal of Mixed Methods Research, 3 (4), 349–370.

Fleck, C., Heilbron, J., Karady, V., & Sapiro, G. (2016). Handbook of indicators of institutionalization of academic disciplines in SSH. Serendipities, Journal for the Sociology and History of the Social Sciences, 1 (1) Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/1 . Accessed 10/10/2018.

Freeman, L. (2004). The development of social network analysis: A study in the sociology of science . Vancouver, Canada: Empirical Press.

Gingras, Y., & Gemme, B. (2006). L’Emprise du champ scientifique sur le champ universitaire et ses effets. Actes de la Recherche en Sciences Sociales, 164 , 51–60.

Goertz, G., & Mahony, J. (2012). A tale of two cultures: Qualitative and quantitative research in the social sciences . Princeton: Princeton University Press.

Greene, J. (2008). Is mixed methods social inquiry a distinctive methodology? Journal of Mixed Methods Research, 2 (1), 7–22.

Guba, E., & Lincoln, Y. (1985). Naturalistic inquiry . Thousand Oaks, CA: Sage.

Heilbron, J., Bedecarré, M., & Timans, R. (2017). European journals in the social sciences and humanities. Serendipities, Journal for the Sociology and History of the Social Sciences, 2 (1), 33–49 Retrieved from http://serendipities.uni-graz.at/index.php/serendipities/issue/view/5 . Accessed 10/10/2018.

Hesse-Biber, S. (2010). Mixed methods research: Merging theory with practice . New York, NY: Guilford Press.

Jahoda, M., Lazarsfeld, P., & Zeisel, H. (1933). Die Arbeitslosen von Marienthal. Psychologische Monographen, 5 .

Johnson, R., & Gray, R. (2010). A history of philosophical and theoretical issues for mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Johnson, R., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1 (2), 112–133.

Law, J. (2004). After method: Mess in social science research . London, UK: Routledge.

Leech, N. (2010). Interviews with the early developers of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Leech, N., & Onwuegbuzie, T. (2009). A typology of mixed methods research designs. Quality & Quantity, 43 (2), 265–275.

Levallois, C., Steinmetz, S., & Wouters, P. (2013). Sloppy data floods or precise social science methodologies? In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge . Cambridge, MA: MIT Press.

Lipset, S., Trow, M., & Coleman, J. (1956). Union democracy: The internal politics of the international typographical union . Glencoe, UK: Free Press.

MacKenzie, D. (1981). Statistics in Britain: 1865–1930: The social construction of scientific knowledge . Edinburgh, UK: Edinburgh University Press.

Morgan, D. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1 (1), 48–76.

Morgan, D. (2018). Living with blurry boundaries: The values of distinguishing between qualitative and quantitative research. Journal of Mixed Methods Research, 12 (3), 268–276.

Mullins, N. (1973). Theories and theory groups in contemporary American sociology . New York, NY: Harper & Row.

Nagy Hesse-Biber, S., & Leavy, P. (Eds.). (2008). Handbook of emergent methods . New York, NY and London, UK: Guilford Press.

Nastasi, B., Hitchcock, J., & Brown, L. (2010). An inclusive framework for conceptualizing mixed method design typologies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Onwuegbuzie, A., Leech, N., & Collins, K. (2011). Toward a new era for conducting mixed analyses: The role of quantitative dominant and qualitative dominant crossover mixed analyses. In M. Williams & P. Vogt (Eds.), Handbook of innovation in social research methods . London, UK: Sage.

Patton, M. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

Plano Clark, V., & Creswell, J. (Eds.). (2007). The mixed methods reader . Thousand Oaks, CA: Sage.

Plano Clark, V., & Ivankova, N. (2016). Mixed methods research: A guide to the field . Thousand Oaks, CA: Sage.

Platt, J. (1996). A history of sociological research methods in America: 1920–1960 . Cambridge, UK: Cambridge University Press.

Plowright, D. (2010). Using mixed methods – Frameworks for an integrated methodology . Thousand Oaks, CA: Sage.

Sandelowski, M. (2014). Unmixing mixed methods research. Research in Nursing & Health, 37 (1), 3–8.

Sapiro, G., Brun, E., & Fordant, C. (2018). The rise of the social sciences and humanities in France: Iinstitutionalization, professionalization and autonomization. In C. Fleck, M. Duller, & V. Karady (Eds.), Shaping human science disciplines: Institutional developments in Europe and beyond . Basingstoke, UK: Palgrave.

Savage, M. (2013). The ‘social life of methods’: A critical introduction. Theory, Culture and Society, 30 (4), 3–21.

Savage, M., & Burrows, R. (2007). The coming crisis of empirical sociology. Sociology, 41 (5), 885–899.

Steinmetz, G. (2016). Social fields, subfields and social spaces at the scale of empires: Explaining the colonial state and colonial sociology. The Sociological Review, 64(2_suppl). 98-123.

Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social and behavioral research . Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (Eds.). (2010a). Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2010b). Overview of contemporary issues in mixed methods. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Chapter   Google Scholar  

Tashakkori, A., & Teddlie, C. (2010c). Epilogue. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

Tashakkori, A., & Teddlie, C. (2011). Mixed methods research: Contemporary issues in an emerging field. In N. Denzin & Y. Lincoln (Eds.), The SAGE handbook of qualitative research (4th ed.). Thousand Oaks, CA: Sage.

Teddlie, C., & Tashakkori, A. (2008). Foundations of mixed methods research – Integrating quantitative and qualitative approaches in the social and behavioral sciences . Thousand Oaks, CA: Sage.

Timans, R. (2015). Studying the Dutch business elite: Relational concepts and methods . Doctoral dissertation, Erasmus University Rotterdam, the Netherlands.

Wacquant, L. (2013). Bourdieu 1993: A case study in scientific consecration. Sociology, 47 (1), 15–29.

Download references

Acknowledgments

This research is part of the Interco-SSH project, funded by the European Union under the 7th Research Framework Programme (grant agreement no. 319974). Johan Heilbron would like to thank Louise and John Steffens, members of the Friends Founders’ Circle, who assisted his stay at the Princeton Institute for Advanced Study in 2017-18 during which he completed his part of the present article.

Author information

Authors and affiliations.

Erasmus Centre for Economic Sociology (ECES), Erasmus University Rotterdam, Rotterdam, Netherlands

Centre for Science and Technology Studies (CWTS), Leiden University, Leiden, Netherlands

Paul Wouters

Erasmus Centre for Economic Sociology (ECES), Rotterdam and Centre européen de sociologie et de science politique de la Sorbonne (CESSP-CNRS-EHESS), Paris, France

Johan Heilbron

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rob Timans .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Timans, R., Wouters, P. & Heilbron, J. Mixed methods research: what it is and what it could be. Theor Soc 48 , 193–216 (2019). https://doi.org/10.1007/s11186-019-09345-5

Download citation

Published : 29 March 2019

Issue Date : 01 April 2019

DOI : https://doi.org/10.1007/s11186-019-09345-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Data, field analysis
• Mixed methods research
• Multiple methods
• Reflexivity
• Sociology of science
• Find a journal
• Publish with us
• Track your research

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals

You are here

• Volume 20, Issue 3
• Mixed methods research: expanding the evidence base
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Allison Shorten 1 ,
• Joanna Smith 2
• 1 School of Nursing , University of Alabama at Birmingham , USA
• 2 Children's Nursing, School of Healthcare , University of Leeds , UK
• Correspondence to Dr Allison Shorten, School of Nursing, University of Alabama at Birmingham, 1720 2nd Ave South, Birmingham, AL, 35294, USA; [email protected]; ashorten{at}uab.edu

https://doi.org/10.1136/eb-2017-102699

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

‘Mixed methods’ is a research approach whereby researchers collect and analyse both quantitative and qualitative data within the same study. 1 2 Growth of mixed methods research in nursing and healthcare has occurred at a time of internationally increasing complexity in healthcare delivery. Mixed methods research draws on potential strengths of both qualitative and quantitative methods, 3 allowing researchers to explore diverse perspectives and uncover relationships that exist between the intricate layers of our multifaceted research questions. As providers and policy makers strive to ensure quality and safety for patients and families, researchers can use mixed methods to explore contemporary healthcare trends and practices across increasingly diverse practice settings.

What is mixed methods research?

Mixed methods research requires a purposeful mixing of methods in data collection, data analysis and interpretation of the evidence. The key word is ‘mixed’, as an essential step in the mixed methods approach is data linkage, or integration at an appropriate stage in the research process. 4 Purposeful data integration enables researchers to seek a more panoramic view of their research landscape, viewing phenomena from different viewpoints and through diverse research lenses. For example, in a randomised controlled trial (RCT) evaluating a decision aid for women making choices about birth after caesarean, quantitative data were collected to assess knowledge change, levels of decisional conflict, birth choices and outcomes. 5 Qualitative narrative data were collected to gain insight into women’s decision-making experiences and factors that influenced their choices for mode of birth. 5

In contrast, multimethod research uses a single research paradigm, either quantitative or qualitative. Data are collected and analysed using different methods within the same paradigm. 6 7 For example, in a multimethods qualitative study investigating parent–professional shared decision-making regarding diagnosis of suspected shunt malfunction in children, data collection included audio recordings of admission consultations and interviews 1 week post consultation, with interactions analysed using conversational analysis and the framework approach for the interview data. 8

What are the strengths and challenges in using mixed methods?

Selecting the right research method starts with identifying the research question and study aims. A mixed methods design is appropriate for answering research questions that neither quantitative nor qualitative methods could answer alone. 4 9–11 Mixed methods can be used to gain a better understanding of connections or contradictions between qualitative and quantitative data; they can provide opportunities for participants to have a strong voice and share their experiences across the research process, and they can facilitate different avenues of exploration that enrich the evidence and enable questions to be answered more deeply. 11 Mixed methods can facilitate greater scholarly interaction and enrich the experiences of researchers as different perspectives illuminate the issues being studied. 11

The process of mixing methods within one study, however, can add to the complexity of conducting research. It often requires more resources (time and personnel) and additional research training, as multidisciplinary research teams need to become conversant with alternative research paradigms and different approaches to sample selection, data collection, data analysis and data synthesis or integration. 11

What are the different types of mixed methods designs?

Mixed methods research comprises different types of design categories, including explanatory, exploratory, parallel and nested (embedded) designs. 2   Table 1 summarises the characteristics of each design, the process used and models of connecting or integrating data. For each type of research, an example was created to illustrate how each study design might be applied to address similar but different nursing research aims within the same general nursing research area.

• View inline

Types of mixed methods designs*

What should be considered when evaluating mixed methods research?

When reading mixed methods research or writing a proposal using mixed methods to answer a research question, the six questions below are a useful guide 12 :

Does the research question justify the use of mixed methods?

Is the method sequence clearly described, logical in flow and well aligned with study aims?

Is data collection and analysis clearly described and well aligned with study aims?

Does one method dominate the other or are they equally important?

Did the use of one method limit or confound the other method?

When, how and by whom is data integration (mixing) achieved?

For more detail of the evaluation guide, refer to the McMaster University Mixed Methods Appraisal Tool. 12 The quality checklist for appraising published mixed methods research could also be used as a design checklist when planning mixed methods studies.

• Elliot AE , et al
• Creswell JW ,
• Plano ClarkV L
• Greene JC ,
• Caracelli VJ ,
• Ivankova NV
• Shorten A ,
• Shorten B ,
• Halcomb E ,
• Cheater F ,
• Bekker H , et al
• Tashakkori A ,
• Creswell JW
• 12. ↵ National Collaborating Centre for Methods and Tools . Appraising qualitative, quantitative, and mixed methods studies included in mixed studies reviews: the MMAT . Hamilton, ON : BMJ Publishing Group , 2015 . http://www.nccmt.ca/resources/search/232 (accessed May 2017) .

Competing interests None declared.

Provenance and peer review Commissioned; internally peer reviewed.

Read the full text or download the PDF:

• Research article
• Open access
• Published: 24 September 2018

A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults

• Gabrielle Lindsay-Smith   ORCID: orcid.org/0000-0003-3864-1412 1 ,
• Grant O’Sullivan 1 ,
• Rochelle Eime 1 , 2 ,
• Jack Harvey 1 , 2 &
• Jannique G. Z. van Uffelen 1 , 3  

BMC Geriatrics volume  18 , Article number:  226 ( 2018 ) Cite this article

74k Accesses

31 Citations

15 Altmetric

Metrics details

Social wellbeing factors such as loneliness and social support have a major impact on the health of older adults and can contribute to physical and mental wellbeing. However, with increasing age, social contacts and social support typically decrease and levels of loneliness increase. Group social engagement appears to have additional benefits for the health of older adults compared to socialising individually with friends and family, but further research is required to confirm whether group activities can be beneficial for the social wellbeing of older adults.

This one-year longitudinal mixed methods study investigated the effect of joining a community group, offering a range of social and physical activities, on social wellbeing of adults with a mean age of 70. The study combined a quantitative survey assessing loneliness and social support ( n  = 28; three time-points, analysed using linear mixed models) and a qualitative focus group study ( n  = 11, analysed using thematic analysis) of members from Life Activities Clubs Victoria, Australia.

There was a significant reduction in loneliness ( p  = 0.023) and a trend toward an increase in social support ( p  = 0.056) in the first year after joining. The focus group confirmed these observations and suggested that social support may take longer than 1 year to develop. Focus groups also identified that group membership provided important opportunities for developing new and diverse social connections through shared interest and experience. These connections were key in improving the social wellbeing of members, especially in their sense of feeling supported or connected and less lonely. Participants agreed that increasing connections was especially beneficial following significant life events such as retirement, moving to a new house or partners becoming unwell.

Conclusions

Becoming a member of a community group offering social and physical activities may improve social wellbeing in older adults, especially following significant life events such as retirement or moving-house, where social network changes. These results indicate that ageing policy and strategies would benefit from encouraging long-term participation in social groups to assist in adapting to changes that occur in later life and optimise healthy ageing.

Peer Review reports

Ageing population and the need to age well

Between 2015 and 2050 it is predicted that globally the number of adults over the age of 60 will more than double [ 1 ]. Increasing age is associated with a greater risk of chronic illnesses such as cardio vascular disease and cancer [ 2 ] and reduced functional capacity [ 3 , 4 ]. Consequently, an ageing population will continue to place considerable pressure on the health care systems.

However, it is also important to consider the individuals themselves and self-perceived good health is very important for the individual wellbeing and life-satisfaction of older adults [ 5 ]. The terms “successful ageing” [ 6 ] and “healthy ageing” [ 5 ] have been used to define a broader concept of ageing well, which not only includes factors relating to medically defined health but also wellbeing. Unfortunately, there is no agreed definition for what exactly constitutes healthy or successful ageing, with studies using a range of definitions. A review of 28 quantitative studies found that successful ageing was defined differently in each, with the majority only considering measures of disability or physical functioning. Social and wellbeing factors were included in only a few of the studies [ 7 ].

In contrast, qualitative studies of older adults’ opinions on successful ageing have found that while good physical and mental health and maintaining physical activity levels are agreed to assist successful ageing, being independent or doing something of value, acceptance of ageing, life satisfaction, social connectedness or keeping socially active were of greater importance [ 8 , 9 , 10 ].

In light of these findings, the definition that is most inclusive is “healthy ageing” defined by the World Health Organisation as “the process of developing and maintaining the functional ability (defined as a combination of intrinsic capacity and physical and social environmental characteristics), that enables well-being in older age” (p28) [ 5 ].This definition, and those provided in the research of older adults’ perceptions of successful ageing, highlight social engagement and social support as important factors contributing to successful ageing, in addition to being important social determinants of health [ 11 , 12 ].

Social determinants of health, including loneliness and social support, are important predictors of physical, cognitive and mental health and wellbeing in adults [ 12 ] and older adults [ 13 , 14 , 15 ]. Loneliness is defined as a perception of an inadequacy in the quality or quantity of one’s social relationships [ 16 ]. Social support, has various definitions but generally it relates to social relationships that are reciprocal, accessible and reliable and provide any or a combination of supportive resources (e.g. emotional, information, practical) and can be measured as perceived or received support [ 17 ]. These types of social determinants differ from those related to inequality (health gap social determinants) and are sometimes referred to as ‘social cure’ social determinants [ 11 ]. They will be referred to as ‘social wellbeing’ outcome measures in this study.

Unfortunately, with advancing age, there is often diminishing social support, leading to social isolation and loneliness [ 18 , 19 ]. Large nationally representative studies of adults and older adults reported that social activity predicted maintenance or improvement of life satisfaction as well as physical activity levels [ 20 ], however older adults spent less time in social activity than middle age adults.

Social wellbeing and health

A number of longitudinal studies have found that social isolation for older adults is a significant predictor of mortality and institutionalisation [ 21 , 22 , 23 ]. A meta-analysis by Holt-Lunstadt [ 12 ] reported that social determinants of health, including social integration and social support (including loneliness and lack of perceived social support) to be equal to, or a greater risk to mortality as common behavioural risk factors such as smoking, physical inactivity and obesity. Loneliness is independently associated with poor physical and mental health in the general population, and especially in older adults [ 13 , 14 , 15 ]. Adequate perceived social support has also been consistently associated with improved mental and physical health in both general and older adults [ 20 , 24 , 25 , 26 , 27 , 28 , 29 ]. The mechanism suggested for this association is that social support buffers the negative impacts of stressful situations and life events [ 30 ]. The above research demonstrates the benefit of social engagement for older adults; in turn this highlights the importance of strategies that reduce loneliness and improve social support and social connectedness for older adults.

Socialising in groups seems to be especially important for the health and wellbeing of older adults who may be adjusting to significant life events [ 26 , 31 , 32 , 33 ]. This is sometimes referred to as social engagement or social companionship [ 26 , 30 , 31 ]. It seems that the mechanism enabling such health benefits with group participation is through strengthening of social identification, which in turn increases social support [ 31 , 34 , 35 ]. Furthermore, involvement in community groups can be a sustainable strategy to reduce loneliness and increase social support in older adults, as they are generally low cost and run by volunteers [ 36 , 37 , 38 , 39 ].

Despite the demonstrated importance of social factors for successful ageing and the established risk associated with reduced social engagement as people age, few in-depth studies have longitudinally investigated the impact of community groups on social wellbeing. For example, a non-significant increase in social support and reduction in depression was found in a year-long randomised controlled trial conducted in senior centres in Norway with lonely older adults in poor physical and mental health [ 37 ]. Some qualitative studies have reported that community groups and senior centres can contribute to fun and socialisation for older adults, however social wellbeing was not the primary focus of the studies [ 38 , 40 , 41 ]. Given that social wellbeing is a broad and important area for the health and quality of life in older adults, an in-depth study is warranted to understand how it can be maximised in older adults. This mixed methods case study of an existing community aims to: i) examine whether loneliness and social support of new members of Life Activities Clubs (LACs) changes in the year after joining and ii) conduct an in-depth exploration of how social wellbeing changes in new and longer-term members of LACs.

A mixed methods study was chosen as the design for this research to enable an in-depth exploration of how loneliness and social support may change as a result of joining a community group. A case study was conducted using a concurrent mixed-methods design, with a qualitative component giving context to the quantitative results. Where the survey focused on the impact of group membership on social support and loneliness, the focus groups were an open discussion of the benefits in the lived context of LAC membership. The synthesis of the two sections of the study was undertaken at the time of interpretation of the results [ 42 ].

The two parts of our study were as follows:

a longitudinal survey (three time points over 1 year: baseline, 6 and 12 months). This part of the study formed the quantitative results;

a focus group study of members of the same organisation (qualitative).

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]) All participants provided informed consent to partake in the study prior to undertaking the first survey or focus group.

Setting and participants

Life activities clubs victoria.

Life Activities Clubs Victoria (LACVI) is a large not-for-profit group with 23 independently run Life Activities Clubs (LACs) based in both rural and metropolitan Victoria. It has approximately 4000 members. The organisation was established to assist in providing physical, social and recreational activities as well as education and motivational support to older adults managing significant change in their lives, especially retirement.

Eighteen out of 23 LAC clubs agreed to take part in the survey study. During the sampling period from May 2014 to December 2016, new members from the participating clubs were given information about the study and invited to take part. Invitations took place in the form of flyers distributed with new membership material.

Inclusion/ exclusion criteria

Community-dwelling older adults who self-reported that they could walk at least 100 m and who were new members to LACVI and able to complete a survey in English were eligible to participate. New members were defined as people who had never been members of LACVI or who had not been members in the last 2 years.

To ensure that the cohort of participants were of a similar functional level, people with significant health problems limiting them from being able to walk 100 m were excluded from participating in the study.

Once informed consent was received, the participants were invited to complete a self-report survey in either paper or online format (depending on preference). This first survey comprised the baseline data and the same survey was completed 6 months and 12 months after this initial time point. Participants were sent reminders if they had not completed each survey more than 2 weeks after each was delivered and then again 1 week later.

Focus groups

Two focus groups (FGs) were conducted with new and longer-term members of LACs. The first FG ( n  = 6) consisted of members who undertook physical activity in their LAC (e.g. walking groups, tennis, cycling). The second FG ( n  = 5) consisted of members who took part in activities with a non-physical activity (PA) focus (e.g. book groups, social groups, craft or cultural groups). LACs offer both social and physical activities and it was important to the study to capture both types of groups, but they were kept separate to assist participants in feeling a sense of commonality with other members and improving group dynamic and participation in the discussions [ 43 ]. Of the people who participated in the longitudinal survey study, seven also participated in the FGs.

The FG interviews were facilitated by one researcher (GLS) and notes around non-verbal communication, moments of divergence and convergence amongst group members, and other notable items were taken by a second researcher (GOS). Both researchers wrote additional notes after the focus groups and these were used in the analysis of themes. Focus groups were recorded and later transcribed verbatim by a professional transcriptionist, including identification of each participant speaking. One researcher (GLS) reviewed each transcription to check for any errors and made any required modifications before importing the transcriptions into NVivo for analysis. The transcriber identified each focus group participant so themes for individuals or other age or gender specific trends could be identified.

Dependent variables

• Social support

Social support was assessed using the Duke–UNC Functional Social support questionnaire [ 44 ]. This scale specifically measures participant perceived functional social support in two areas; i) confidant support (5 questions; e.g. chances to talk to others) and ii) affective support (3 questions; e.g. people who care about them). Participants rated each component of support on a 5-item likert scale between ‘much less than I would like’ (1 point) to ‘as much as I would like’ (5 points). The total score used for analysis was the mean of the eight scores (low social support = 1, maximum social support = 5). Construct validity, concurrent validity and discriminant validity are acceptable for confidant and affective support items in the survey in the general population [ 44 ].

Loneliness was measured using the de Jong Gierveld and UCLA-3 item loneliness scales developed for use in many populations including older adults [ 45 ]. The 11-item de Jong Gierveld loneliness scale (DJG loneliness) [ 46 ] is a multi-dimensional measure of loneliness and contains five positively worded and six negatively worded items. The items fall into four subscales; feelings of severe loneliness, feelings connected with specific problem situations, missing companionship, feelings of belongingness. The total score is the sum of the items scores (i.e. 11–55): 11 is low loneliness and 55 is severe loneliness. Self-administered versions of this scale have good internal consistency (> = 0.8) and inter-item homogeneity and person scalability that is as good or better than when conducted as face-to face interviews. The validity and reliability for the scale is adequate [ 47 ]. The UCLA 3-item loneliness scale consists of three questions about how often participants feel they lack companionship, feel left out and feel isolated. The responses are given on a three-point scale ranging from hardly ever (1) to often (3). The final score is the sum of these three items with the range being from lowest loneliness (3) to highest loneliness (9). Reliability of the scale is good, (alpha = 0.72) as are discriminant validity and internal consistency [ 48 ]. The scale is commonly used to measure loneliness with older adults ([ 49 ] – review), [ 50 , 51 ].

Sociodemographic variables

The following sociodemographic characteristics were collected in both the survey and the focus groups: age, sex, highest level of education, main life occupation [ 52 ], current employment, ability to manage on income available, present marital status, country of birth, area of residence [ 53 ]. They are categorised as indicated in Table  2 .

Health variables

The following health variables were collected: Self-rated general health (from SF-12) [ 54 ] and Functional health (ability to walk 100 m- formed part of the inclusion criteria) [ 55 ]. See Table 2 for details about the categories of these variables.

The effects of becoming a member on quantitative outcome variables (i.e. Social support, DJG loneliness and UCLA loneliness) were analysed using linear mixed models (LMM). LMM enabled testing for the presence of intra-subject random effects, or equivalently, correlation of subjects’ measures over time (baseline, 6-months and 12 months). Three correlation structures were examined: independence (no correlation), compound symmetry (constant correlation of each subjects’ measures over the three time points) and autoregressive (correlation diminishing with increase in spacing in time). The best fitting correlation structure was compound symmetry; this is equivalent to a random intercept component for each subject. The LMM incorporated longitudinal trends over time, with adjustment for age as a potential confounder. Statistical analyses were conducted using SPSS for windows (v24).

UCLA loneliness and social support residuals were not normally distributed and these scales were Log10 transformed for statistical analysis.

Analyses were all adjusted for age, group attendance (calculated as average attendance at 6 and 12 months) and employment status at baseline (Full-time, Part-time, not working).

Focus group transcripts were analysed using thematic analysis [ 56 , 57 ], a flexible qualitative methodology that can be used with a variety of epistemologies, approaches and analysis methods [ 56 ]. The transcribed data were analysed using a combination of theoretical and inductive thematic analysis [ 56 ]. It was theorised that membership in a LAC would assist with social factors relating to healthy ageing [ 5 ], possibly through a social identity pathway [ 58 ], although we wanted to explore this. Semantic themes were drawn from these codes in order to conduct a pragmatic evaluation of the LACVI programs [ 56 ]. Analytic rigour in the qualitative analysis was ensured through source and analyst triangulation. Transcriptions were compared to notes taken during the focus groups by the researchers (GOS and GLS). In addition, Initial coding and themes (by GLS) were checked by a second researcher (GOS) and any disagreements regarding coding and themes were discussed prior to finalisation of codes and themes [ 57 ].

Sociodemographic and health characteristics of the 28 participants who completed the survey study are reported in Table  1 . The mean age of the participants was 66.9 and 75% were female. These demographics are representative of the entire LACVI membership. Education levels varied, with 21% being university educated, and the remainder completing high school or technical certificates. Two thirds of participants were not married. Some sociodemographic characteristics changed slightly at 6 and 12 months, mainly employment (18% in paid employment at baseline and 11% at 12-months) and ability to manage on income (36% reporting trouble managing on their income at baseline and 46% at 12 months). Almost 90% of the participants described themselves as being in good-excellent health.

Types of activities

There were a variety of types of activities that participants took part in: physical activities such as walking groups ( n  = 7), table tennis ( n  = 5), dancing class ( n  = 2), exercise class ( n  = 1), bowls ( n  = 2), golf ( n  = 3), cycling groups ( n  = 1) and non-physical leisure activities such as art and literature groups ( n  = 5), craft groups ( n  = 5), entertainment groups ( n  = 12), food/dine out groups ( n  = 18) and other sedentary leisure activities (e.g. mah jong, cards),( n  = 4). A number of people took part in more than one activity.

Frequency of attendance at LACVI and changes in social wellbeing

At six and 12 months, participants indicated how many times in the last month they attended different types of activities at their LAC. Most participants maintained the same frequency of participation over both time points. Only four people participated more frequently at 12 than at 6 months and nine reduced participation levels. The latter group included predominantly those who reduced from more than two times per week at 6 months to 2×/week at 6 months to one to two times per week ( n  = 5) or less than one time per week ( n  = 2) at 12 months. Average weekly club attendance at six and 12 months was included as a covariate in the statistical model.

Outcome measures

Overall, participants reported moderate social support and loneliness levels at baseline (See Table 2 ). Loneliness, as measured by both scales, reduced significantly over time. There was a significant effect of time on the DJG loneliness scores (F (2, 52) = 3.83, p  = 0.028), with Post-Hoc analysis indicating a reduction in DJG loneliness between baseline and 12 months ( p  = 0.008). UCLA loneliness scores (transformed variable) also changed significantly over time (F (2, 52) = 4.08, p  = 0.023). Post hoc tests indicated a reduction in UCLA loneliness between baseline and 6 months ( p  = 0.007). There was a small non-significant increase in social support (F (2, 53) =2.88, p  = 0.065) during the first year of membership (see Table 2 and Figs. 1 and 2 ).

DJG loneliness for all participants over first year of membership at LAC club ( n  = 28).

*Represents significant difference compared to baseline ( p  < 0.01)

UCLA loneliness score for all participants over first year of membership at LAC club ( n  = 28).

*Indicates log values of the variable at 6-months were significantly different from baseline ( p  < 0.01)

In total, 11 participants attended the two focus groups, six people who participated in PA clubs (four women) and five who participated in social clubs (all women). All focus group participants were either retired ( n  = 9) or semi-retired ( n  = 2). The mean age of participants was 67 years (see Table 2 for further details). Most of the participants (82%) had been members of a LAC for less than 2 years and two females in the social group had been members of LAC clubs for 5 and 10 years respectively.

Analysis of the focus group transcripts identified two themes relating to social benefits of group participation; i) Social resources and ii) Social wellbeing (see Fig. 3 ). Group discussion suggested that membership of a LAC provides access to more social resources through greater and diverse social contact and opportunity. It is through this improvement in social resources that social wellbeing may improve.

Themes arising from focus group discussion around the benefits of LAC membership

Social resources

The social resources theme referred to an increase in the availability and variety of social connections that resulted from becoming a member of a LAC. The social nature of the groups enabled an expansion and diversification of members’ social network and improved their sense of social connectedness. There was widespread agreement in both the focus groups that significant life events, especially retirement, illness or death of spouse and moving house changes one’s social resources. Membership of the LAC had benefits especially at these times and these events were often motivators to join such a club. Most participants found that their social resources declined after retirement and even felt that they were grieving for the loss of their work.

“ I just saw work as a collection of, um, colleagues as opposed to friends. I had a few good friends there. Most were simply colleagues or acquaintances …. [interviewer- Mmm.] ..Okay, you’d talk to them every day. You’d chatter in the kitchen, oh, pass banter back and forth when things are busy or quiet, but... Um, in terms of a friendship with those people, like going to their home, getting to know them, doing other things with them, very few. But what I did miss was the interaction with other people. It had simply gone….. But, yeah, look, that, the, yeah, that intervening period was, oh, a couple of months. That was a bit tough…. But in that time the people in LAC and the people in U3A…. And the other dance group just drew me into more things. Got to know more people. So once again, yeah, reasonable group of acquaintances.” (Male, PAFG)

Group members indicated general agreement with these two responses, however one female found she had a greater social life following retirement due to the busy nature of her job.

Within the social resources theme, three subthemes were identified, i) Opportunity for social connectedness, ii) Opportunity for friendships, and iii) Opportunity for social responsibility/leadership . Interestingly, these subthemes were additional to the information gathered in the survey. This emphasises the power of the inductive nature of the qualitative exploration employed in the focus groups to broaden the knowledge in this area.

The most discussed and expanded subtheme in both focus groups was Opportunity for social connectedness , which arose through developing new connections, diversifying social connections, sharing interests and experiences with others and peer learning. Participants in both focus groups stated that being a member of LAC facilitated their socialising and connecting with others to share ideas, skills and to do activities with, which was especially important through times of significant life events. Furthermore, participants in each of the focus groups valued developing diverse connections:

“ Yeah, I think, as I said, I finished up work and I, and I had more time for wa-, walking. So I think a, in meeting, in going to this group which, I saw this group of women but then someone introduced me to them. They were just meeting, just meeting a new different set of people, you know? As I said, my work people and these were just a whole different group of women, mainly women. There’s not many men. [Interviewer: Yes.]….. Although our leader is a man, which is ironic and is about, this man out in front and there’s about 20 women behind him, but, um, so yeah, and people from different walks of life and different nationalities there which I never knew in my work life, so yeah. That’s been great. So from that goes on other things, you know, you might, uh, other activities and, yeah, people for coffee and go to the pictures or something, yeah. That’s great.” (Female, PAFG)

Simply making new connections was the most widely discussed aspect related to the opportunity for social connectedness subtheme, with all participants agreeing that this was an important benefit of participation in LAC groups.

“Well, my experience is very similar to everybody else’s…….: I, I went from having no social life to a social life once I joined a group.” (Female, PAFG)

There was agreement in both focus groups that these initial new connections made at a LAC are strengthened through development of deeper personal connections with others who have similar demographics and who are interested in the same activities. This concurs with the Social Identity Theory [ 58 ] discussed previously.

“and I was walking around the lake in Ballarat, like wandering on my own. I thought, This is ridiculous. I mean, you’ve met all those groups of women coming the opposite way, so I found out what it was all about, so I joined, yeah. So that’s how I got into that.[ Interviewer: Yeah.] Basically sick of walking round the lake on my own. [Interviewer: Yeah, yeah.] So that’s great. It’s very social and they have coffee afterwards which is good.” (female, PAFG)

The subtheme Opportunity for development of friendships describes how, for some people, a number of LAC members have progressed from being just initial social connections to an established friendship. This signifies the strength of the connections that may potentially develop through LAC membership. Some participants from each group mentioned friendships developing, with slightly more discussion of this seen in the social group.

“we all have a good old chat, you know, and, and it’s all about friendship as well.” (female, SocialFG)

The subtheme Opportunity for social responsibility or leadership was mentioned by two people in the active group, however it was not brought up in the social group. This opportunity for leadership is linked with the development of a group identity and desiring to contribute meaningfully to a valued group.

“with our riding group, um, you, a leader for probably two rides a year so you’ve gotta prepare for it, so some of them do reccie rides themselves, so, um, and also every, uh, so that’s something that’s, uh, a responsibility.” (male, PAFG)

Social wellbeing

The social resources described above seem to contribute to a number of social, wellbeing outcomes for participants. The sub themes identified for Social wellbeing were , i) Increased social support, ii) Reduced loneliness, iii) Improved home relationships and iv) Improved social skills.

Increased social support

Social support was measured quantitatively in the survey (no significant change over time for new members) and identified as a benefit of LAC membership during the focus group discussions. However, only one of the members of the active group mentioned social support directly.

‘it’s nice to be able to pick up the phone and share your problem with somebody else, and that’s come about through LAC. ……‘Cos before that it was through, with my family (female, PAFG)

There was some agreement amongst participants of the PA group that they felt this kind of support may develop in time but most of them had been members for less than 2 years.

“[Interviewer: Yeah. Does anyone else have that experience? (relating to above quote)]” There is one lady but she’s actually the one that I joined with anyway. [Interviewer: Okay.] But I, I feel there are others that are definitely getting towards that stage. It’s still going quite early days. (female1, PAFG) [Interviewer: I guess it’s quite early for some of you, yeah.] “yeah” (female 2, PAFG)

Social support through sharing of skills was mentioned by one participant in the social group also, with agreement indicated by most of the others in the social focus group.

Discussion in the focus groups also touched on the subthemes Reduced loneliness and Improved home relationships, which were each mentioned by one person. And focus groups also felt that group membership Improved social skills through opening up and becoming more approachable (male, PAFG) or enabling them to become more accepting of others’ who are different (general agreement in Social FG).

This case study integrated results from a one-year longitudinal survey study and focus group discussions to gather rich information regarding the potential changes in social wellbeing that older adults may experience when joining community organisations offering group activities. The findings from this study indicate that becoming a member of such a community organisation can be associated with a range of social benefits for older adults, particularly related to reducing loneliness and maintaining social connections.

Joining a LAC was associated with a reduction in loneliness over 1 year. This finding is in line with past group-intervention studies where social activity groups were found to assist in reducing loneliness and social isolation [ 49 ]. This systematic review highlighted that the majority of the literature explored the effectiveness of group activity interventions for reducing severe loneliness or loneliness in clinical populations [ 49 ]. The present study extends this research to the general older adult population who are not specifically lonely and reported to be of good general health, rather than a clinical focus. Our findings are in contrast to results from an evaluation of a community capacity-building program aimed at reducing social isolation in older adults in rural Australia [ 59 ]. That program did not successfully reduce loneliness or improve social support. The lack of change from pre- to post-program in that study was reasoned to be due to sampling error, unstandardised data collection, and changes in sample characteristics across the programs [ 59 ]. Qualitative assessment of the same program [ 59 ] did however suggest that participants felt it was successful in reducing social isolation, which does support our findings.

Changes in loneliness were not a main discussion point of the qualitative component of the current study, however some participants did express that they felt less lonely since joining LACVI and all felt they had become more connected with others. This is not so much of a contrast in results as a potential situational issue. The lack of discussion of loneliness may have been linked to the common social stigma around experiencing loneliness outside certain accepted circumstances (e.g. widowhood), which may lead to underreporting in front of others [ 45 ].

Overall, both components of the study suggest that becoming a member of an activity group may be associated with reductions in loneliness, or at least a greater sense of social connectedness. In addition to the social nature of the groups and increased opportunity for social connections, another possible link between group activity and reduced loneliness is an increased opportunity for time out of home. Previous research has found that more time away from home in an average day is associated with lower loneliness in older adults [ 60 ]. Given the significant health and social problems that are related to loneliness and social isolation [ 13 , 14 , 15 ], the importance of group involvement for newly retired adults to prevent loneliness should be advocated.

In line with a significant reduction in loneliness, there was also a trend ( p  = 0.056) toward an increase in social support from baseline to 12 months in the survey study. Whilst suggestive of a change, it is far less conclusive than the findings for loneliness. There are a number of possible explanations for the lack of statistically significant change in this variable over the course of the study. The first is the small sample size, which would reduce the statistical power of the study. It may be that larger studies are required to observe changes in social support, which are possibly only subtle over the course of 1 year. This idea is supported by a year-long randomised controlled trial with 90 mildly-depressed older adults who attended senior citizen’s club in Norway [ 37 ]. The study failed to see any change in general social support in the intervention group compared to the control over 1 year. Additional analysis in that study suggested that people who attended the intervention groups more often, tended to have greater increases in SS ( p  = 0.08). The researchers stated that the study suffered from significant drop-out rates and low power as a result. In this way, it was similar to our findings and suggests that social support studies require larger numbers than we were able to gain in this early exploratory study. Another possible reason for small changes in SS in the current study may be the type of SS measured. The scale used gathered information around functional support or support given to individuals in times of need. Maybe it is not this type of support that changes in such groups but more specific support such as task-specific support. It has been observed in other studies and reviews that task-specific support changes as a result of behavioural interventions (e.g. PA interventions) but general support does not seem to change in the time frames often studied [ 61 , 62 , 63 ].

There were many social wellbeing benefits such as increased social connectivity identified in focus group discussion, but the specific theme of social support was rarely mentioned. It may be that general social support through such community groups may take longer than 1 year to develop. There is evidence that strong group ties are sequentially positively associated between social identification and social support [ 34 ], suggesting that the connections formed through the groups may lead increased to social support from group members in the future. This is supported by results from the focus group discussions, where one new member felt she could call on colleagues she met in her new group. Other new members thought it was too soon for this support to be available, but they could see the bonds developing.

Other social wellbeing changes

In addition to social support and loneliness that were the focus of the quantitative study, the focus group discussions uncovered a number of other benefits of group membership that were related to social wellbeing (see Fig. 3 ). The social resources theme was of particular interest because it reflected some of the mechanisms that appeared enable social wellbeing changes as a result of being a member of a LAC but were not measured in the survey. The main social resources relating to group membership that were mentioned in the focus groups were social connectedness, development of friendships and opportunity for social responsibility or leadership. As mentioned above, there was wide-spread discussion within the focus groups of the development of social connections through the clubs. Social connectedness is defined as “the sense of belonging and subjective psychological bond that people feel in relation to individuals and groups of others.” ([ 25 ], pp1). As well as being an important predecessor of social support, greater social connectedness has been found to be highly important for the health of older adults, especially cognitive and mental health [ 26 , 32 , 34 , 35 , 64 ]. One suggested theory for this health benefit is that connections developed through groups that we strongly identify with are likely to be important for the development of social identity [ 34 ], defined by Taifel as: “knowledge that [we] belong to certain social groups together with some emotional and value significance to [us] of this group membership” (Tajfel, 1972, p. 31 in [ 58 ] p 2). These types of groups to which we identify may be a source of “personal security, social companionship, emotional bonding, intellectual stimulation, and collaborative learning and……allow us to achieve goals.” ([ 58 ] p2) and an overall sense of self-worth and wellbeing. There was a great deal of discussion relating to the opportunity for social connectedness derived through group membership being particularly pertinent following a significant life event such as moving to a new house or partners becoming unwell or dying and especially retirement. This change in their social circumstance is likely to have triggered the need to renew their social identity by joining a community group. Research with university students has shown that new group identification can assist in transition for university students who have lost their old groups of friends because of starting university [ 65 ]. In an example relevant to older adults, maintenance or increase in number of group memberships at the time of retirement reduced mortality risk 8 years later compared to people who reduce their number of group activities in a longitudinal cohort study [ 66 ]. This would fit with the original Activity Theory of ageing; whereby better ageing experience is achieved when levels of social participation are maintained, and role replacement occurs when old roles (such as working roles) must be relinquished [ 67 ]. These connections therefore appear to assist in maintaining resilience in older adults defined as “the ability to maintain or improve a level of functional ability (a combination of intrinsic physical and mental capacity and environment) in the face of adversity” (p29, [ 5 ]). Factors that were mentioned in the focus groups as assisting participants in forming connections with others were shared interest, learning from others, and a fun and accepting environment. It was not possible to assess all life events in the survey study. However, since the discussion from the focus groups suggested this to be an important motivator for joining clubs and potentially a beneficial time for joining them, it would be worth exploring in future studies.

Focus group discussion suggested that an especially valuable time for joining such clubs was around retirement, to assist with maintaining social connectivity. The social groups seem to provide social activity and new roles for these older adults at times of change. It is not necessarily important for all older adults but maybe these ones identify themselves as social beings and therefore this maintenance of social connection helps to continue their social role. Given the suggested importance of social connectivity gained through this organisation, especially at times of significant life events, it would valuable to investigate this further in future and consider encouragement of such through government policy and funding. The majority of these types of clubs exist for older adults in general, but this study emphasises the need for groups such as these to target newly retired individuals specifically and to ensure that they are not seen as ‘only for old people’.

Strengths and limitations

The use of mixed –methodologies, combining longitudinal survey study analysed quantitatively, with a qualitative exploration through focus group discussions and thematic analysis, was a strength of the current study. It allowed the researchers to not only examine the association between becoming a member of a community group on social support and loneliness over an extended period, but also obtain a deeper understanding of the underlying reasons behind any associations. Given the variability of social support definitions in research [ 17 ] and the broad area of social wellbeing, it allowed for open exploration of the topic, to understand associations that may exist but would have otherwise been missed. Embedding the research in an existing community organisation was a strength, although with this also came some difficulties with recruitment. Voluntary coordination of the community groups meant that informing new members about the study was not always feasible or a priority for the volunteers. In addition, calling for new members was innately challenging because they were not yet committed to the club fully. This meant that so some people did not want to commit to a year-long study if they were not sure how long they would be a member of the club. This resulted in slow recruitment and a resulting relatively low sample size and decreased power to show significant statistical differences, which is a limitation of the present study. However, the use of Linear Mixed Models for analysis of the survey data was a strength because it was able to include all data in the analyses and not remove participants if one time point of data was missing, as repeated measures ANOVAs would do. The length of the study (1 year) is another strength, especially compared to previous randomised controlled studies that are typically only 6–16 weeks in length. Drop-out rate in the current study is very low and probably attributable to the benefits of working with long-standing organisations.

The purpose of this study was to explore in detail whether there are any relationships between joining existing community groups for older adults and social wellbeing. The lack of existing evidence in the field meant that a small feasibility-type case study was a good sounding-board for future larger scale research on the topic, despite not being able to answer questions of causality. Owing to the particularistic nature of case studies, it can also be difficult to generalise to other types of organisations or groups unless there is a great deal of similarity between them [ 68 ]. There are however, other types of community organisations in existence that have a similar structure to LACVI (Seniors centres [ 36 , 40 ], Men’s Sheds [ 38 ], University of the Third Age [ 34 , 69 ], Japanese salons [ 70 , 71 ]) and it may be that the results from this study are transferable to these also. This study adds to the literature around the benefits of joining community organisations that offer social and physical activities for older adults and suggests that this engagement may assist with reducing loneliness and maintaining social connection, especially around the time of retirement.

Directions for future research

Given that social support trended toward a significant increase, it would be useful to repeat the study on a larger scale in future to confirm this. Either a case study on a similar but larger community group or combining a number of community organisations would enable recruitment of more participants. Such an approach would also assist in assessing the generalisability of our findings to other community groups. Given that discussions around social benefits of group membership in the focus groups was often raised in conjunction with the occurrence of significant life events, it would be beneficial to include a significant life event scale in any future studies in this area. The qualitative results also suggest that it would be useful to investigate whether people who join community groups in early years post retirement gain the same social benefits as those in later stages of retirement. Studies investigating additional health benefits of these community groups such as physical activity, depression and general wellbeing would also be warranted.

With an ageing population, it is important to investigate ways to enable older adults to age successfully to ensure optimal quality of life and minimisation of health care costs. Social determinants of health such as social support, loneliness and social contact are important contributors to successful ageing through improvements in cognitive health, quality of life, reduction in depression and reduction in mortality. Unfortunately, older adults are at risk of these social factors declining in older age and there is little research investigating how best to tackle this. Community groups offering a range of activities may assist by improving social connectedness and social support and reducing loneliness for older adults. Some factors that may assist with this are activities that encourage sharing interests, learning from others, and are conducted in a fun and accepting environment. Such groups may be particularly important in developing social contacts for newly retired individuals or around other significant life events such as moving or illness of loved ones. In conclusion, ageing policy and strategies should emphasise participation in community groups especially for those recently retired, as they may assist in reducing loneliness and increasing social connections for older adults.

Abbreviations

Focus group

Life Activities Club

Life Activities Clubs Victoria

Linear mixed model

Physical activity

World Health Organisation

United Nations. In: P.D. Department of Economic and Social Affairs, editor. World Population Ageing 2015. New York: United Nations; 2015.

Google Scholar  

World Health Organisation. Global Health and Ageing. 2011 [cited 2014 March 25]; Available from: http://www.who.int/ageing/publications/global_health/en/ .

Balogun JA, et al. Age-related changes in balance performance. Disabil Rehabil. 1994;16(2):58–62.

Article   CAS   Google Scholar  

Singh MAF. Exercise comes of age: rationale and recommendations for a geriatric exercise prescription. J Gerontol Ser A Biol Med Sci. 2002;57(5):M262–82.

Article   Google Scholar  

World Health Organisation. World report on ageing and health. Geneva: World Health Organisation; 2015.

Rowe JW, Kahn RL. Successful Aging1. The Gerontologist. 1997;37(4):433–40.

Depp CA, Jeste DV. Definitions and predictors of successful aging: a comprehensive review of larger quantitative studies. Am J Geriatr Psychiatry. 2006;14(1):6–20.

Song M, Kong E-H. Older adults’ definitions of health: a metasynthesis. Int J Nurs Stud. 2015;52(6):1097–106.

Tate RB, Lah L, Cuddy TE. Definition of successful aging by elderly Canadian males: the Manitoba follow-up study. The Gerontologist. 2003;43(5):735–44.

Phelan EA, et al. Older Adults’ views of “successful aging”—how do they compare with Researchers’ definitions? J Am Geriatr Soc. 2004;52(2):211–6.

Haslam SA, et al. Social cure, what social cure? The propensity to underestimate the importance of social factors for health. Soc Sci Med. 2018;198:14–21.

Holt-Lunstad J, Smith TB, Layton JB. Social relationships and mortality risk: a meta-analytic review. PLoS Med. 2010;7(7):e1000316.

Richard A, et al. Loneliness is adversely associated with physical and mental health and lifestyle factors: results from a Swiss national survey. PLoS One. 2017;12(7):1–18.

Luo Y, et al. Loneliness, health, and mortality in old age: a national longitudinal study. Soc Sci Med. 2012;74(6):907–14.

Luo Y, Waite LJ. Loneliness and mortality among older adults in China. J Gerontol B Psychol Sci Soc Sci. 2014;69(4):633–45.

de Jong Gierveld J, Van Tilburg T, Dykstra PA. Loneliness and social isolation. In: Vangelisti A, Perlman D, editors. Cambridge handbook of personal relationships. Cambridge: Cambridge University Press; 2006. p. 485-500.

Williams P, Barclay L, Schmied V. Defining social support in context: a necessary step in improving research, intervention, and practice. Qual Health Res. 2004;14(7):942–60.

Valtorta N, Hanratty B. Loneliness, isolation and the health of older adults: do we need a new research agenda? J R Soc Med. 2012;105(12):518–22.

Jylhä M. Old age and loneliness: cross-sectional and longitudinal analyses in the Tampere longitudinal study on aging. Can J Aging La Revue can du vieil. 2010;23(2):157–68.

Huxhold O, Miche M, Schüz B. Benefits of having friends in older ages: differential effects of informal social activities on well-being in middle-aged and older adults. J Gerontol B Psychol Sci Soc Sci. 2014;69(3):366–75.

Berkman L, Syme S. Social networks, host resistance and mortality: a nine year follow-up study of alameda county residents. Am J Epidemiol. 1979;185(11):1070–88.

House JS, Landis KR, Umberson D. Social relationships and health. Science. 1988;241(4865):540–5.

Pynnonen K, et al. Does social activity decrease risk for institutionalization and mortality in older people? J Gerontol B Psychol Sci Soc Sci. 2012;67(6):765–74.

Broadhead WE, et al. The epidemiologic evidence for a relationship between social support and health. Am J Epidemiol. 1983;117(5):521–37.

Haslam C, et al. Social connectedness and health. Encyclopedia of geropsychology. 2017:2174–82. https://doi.org/10.1007/978-981-287-080-3_46-1 .

Haslam C, Cruwys T, Haslam SA. “The we’s have it”: evidence for the distinctive benefits of group engagement in enhancing cognitive health in aging. Soc Sci Med. 2014;120:57–66.

Uebelacker LA, et al. Social support and physical activity as moderators of life stress in predicting baseline depression and change in depression over time in the Women’s Health Initiative. Soc Psychiatry Psychiatr Epidemiol. 2013;48(12):1971–82.

Tajvar M, et al. Social support and health of older people in middle eastern countries: a systematic review. Australas J Ageing. 2013;32(2):71–8.

Dalgard OS, Bjork S, Tambs K. Social support, negative life events and mental health. Br J Psychiatry. 1995;166(1):29–34.

Cohen S, Wills TA. Stress, social support, and the buffering hypothesis. Psychol Bull. 1985;98(2):310–57.

Gilmour H. Social participation and the health and well-being of Canadian seniors. Health Rep. 2012;23(4):1B.

Glei DA, et al. Participating in social activities helps preserve cognitive function: an analysis of a longitudinal, population-based study of the elderly. Int J Epidemiol. 2005;34(4):864–71.

Lee SH, Kim YB. Which type of social activities may reduce cognitive decline in the elderly?: a longitudinal population-based study. BMC Geriatr. 2016;16(1):165.

Haslam C, et al. Group ties protect cognitive health by promoting social identification and social support. J Aging Health. 2016;28(2):244–66.

Haslam C, et al. Groups 4 health: evidence that a social-identity intervention that builds and strengthens social group membership improves mental health. J Affect Disord. 2016;194:188–95.

Bøen H. Characteristics of senior Centre users -- and the impact of a group programme on social support and late-life depression. Norsk Epidemiologi. 2012;22(2):261–9.

Bøen H, et al. A randomized controlled trial of a senior Centre group programme for increasing social support and preventing depression in elderly people living at home in Norway. BMC Geriatr. 2012;12:20.

Golding BG. Social, local, and situated: recent findings about the effectiveness of older Men’s informal learning in community contexts. Adult Educ Q. 2011;61(2):103.

Life Activities Clubs. Life Activities Clubs. About Us. . 2014 [cited 2014 January 13, 2014]; Available from: http://www.life.org.au/aboutus .

Hutchinson SL, Gallant KA. Can senior Centres be contexts for aging in third places? J Leis Res. 2016;48(1):50–68.

Millard J. The health of older adults in community activities. Work Older People. 2017;21(2):90–9.

Creswell JW, Plano-Clark VL. Designing and conducting mixed methods research. Thousand Oaks, Calif: SAGE Publications; 2007. p. c2007.

Loeb S, Penrod J, Hupcey J. Focus groups and older adults: tactics for success. J Gerontol Nurs. 2006;32(3):32–8.

PubMed   Google Scholar  

Broadhead W, et al. The Duke-UNC functional social support questionnaire: measurement of social support in family medicine patients. Med Care. 1988;26(7):709–23.

De Jong Gierveld J, Van Tilburg T. Living arrangements of older adults in the Netherlands and Italy: Coresidence values and behaviour and their consequences for loneliness. J Cross Cult Gerontol. 1999;14(1):1–24.

de Jong-Gierveld J, Kamphuls F. The development of a Rasch-type loneliness scale. Appl Psychol Meas. 1985;9(3):289–99.

Tilburg Tv, Leeuw Ed. Stability of scale quality under various data collection procedures: a mode comparison on the ‘De Jong-Gierveld loneliness scale. Int J Public Opin Res. 1991;3(1):69–85.

Hughes ME, et al. A short scale for measuring loneliness in large surveys - results from two population-based studies. Res Aging. 2004;26(6):655–72.

Dickens AP, et al. Interventions targeting social isolation in older people: a systematic review. BMC Public Health. 2011;11:647.

Hawkley LC, et al. Loneliness predicts increased blood pressure: 5-year cross-lagged analyses in middle-aged and older adults. Psychol Aging. 2010;25(1):132.

Netz Y, et al. Loneliness is associated with an increased risk of sedentary life in older Israelis. Aging Ment Health. 2013;17(1):40–7.

Australian Bureau of Statistics. Australian and New Zealand Standard Classification of Occupations, 2013, Version 1.2. Canberra: Australian Bureau of Statistics; 2013.

Australian Bureau of Statistics, Australian Standard Geographical Classification (ASGC). Canberra: Australian Bureau of Statistics; 2011.

Ware JE, Kosinski M, Keller SD. A 12-item short-form health survey - construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.

Sanson-Fisher RW, Perkins JJ. Adaptation and validation of the SF-36 health survey for use in Australia. J Clin Epidemiol. 51(11):961–7.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.

Patton MQ. Qualitative research & evaluation methods. 4 ed. Thousand Oaks, California: Sage Publications; 2015.

Haslam SA, et al. Social identity, health and well-being: an emerging agenda for applied psychology. Appl Psychol. 2009;58(1):1–23.

Bartlett H, et al. Preventing social isolation in later life: findings and insights from a pilot Queensland intervention study. Ageing Soc. 2013;33(07):1167–89.

Petersen J, et al. Time out-of-home and cognitive, physical, and emotional wellbeing of older adults: a longitudinal mixed effects model. PLoS One. 2015;10(10):e0139643.

Lindsay Smith G, et al. The association between social support and physical activity in older adults: a systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56.

Sallis JF, et al. The development of scales to measure social support for diet and exercise behaviors. Prev Med. 1987;16(6):825–36.

Oka R, King A, Young DR. Sources of social support as predictors of exercise adherence in women and men ages 50 to 65 years. Womens Health Res Gender Behav Policy. 1995;1:161–75.

CAS   Google Scholar  

Greenaway KH, et al. From “we” to “me”: group identification enhances perceived personal control with consequences for health and well-being. J Pers Soc Psychol. 2015;109(1):53–74.

Iyer A, et al. The more (and the more compatible) the merrier: multiple group memberships and identity compatibility as predictors of adjustment after life transitions. Br J Soc Psychol. 2009;48(4):707–33.

Steffens NK, et al. Social group memberships in retirement are associated with reduced risk of premature death: evidence from a longitudinal cohort study. BMJ Open. 2016;6(2):e010164.

Havighurst RJ. Successful aging. Gerontol. 1961;1:8–13.

Yin R. Case study research: design and methods. Beverly Hills, CA: Sage publishing; 1994.

Merriam SB, Kee Y. Promoting community wellbeing: the case for lifelong learning for older adults. Adult Educ Q. 2014;64(2):128–44.

Hikichi H, et al. Social interaction and cognitive decline: results of a 7-year community intervention. Alzheimers Dement: Translat Res Clin Interv. 2017;3(1):23–32.

Hikichi H, et al. Effect of a community intervention programme promoting social interactions on functional disability prevention for older adults: propensity score matching and instrumental variable analyses, JAGES Taketoyo study. J Epidemiol Community Health. 2015;69(9):905–10.

Download references

The primary author contributing to this study (GLS) receives PhD scholarship funding from Victoria University. The other authors were funded through salaries at Victoria University.

Availability of data and materials

The datasets generated and/or analysed during the current study are not publicly available due the ethics approval for this study not allowing open access to the individual participant data but are available from the corresponding author on reasonable request.

Author information

Authors and affiliations.

Institute for Health and Sport, Victoria University, Melbourne, Australia

Gabrielle Lindsay-Smith, Grant O’Sullivan, Rochelle Eime, Jack Harvey & Jannique G. Z. van Uffelen

School of Health and Life Sciences, Federation University, Ballarat, Australia

Rochelle Eime & Jack Harvey

Department of Movement Sciences, Physical Activity, Sports and Health Research Group, KU Leuven - University of Leuven, Leuven, Belgium

Jannique G. Z. van Uffelen

You can also search for this author in PubMed   Google Scholar

Contributions

GLS, RE and JVU made substantial contributions to the conception and design of the study. GLS and GOS supervised data collection for the surveys (GLS) and focus groups (GOS and GLS). GLS, GOS, RE, JH and JVU were involved in data analysis and interpretation. All authors were involved in drafting, the manuscript and approved the final version.

Corresponding author

Correspondence to Gabrielle Lindsay-Smith .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval to conduct this study was obtained from the Victoria University Human Research Ethics Committee (HRE14–071 [survey] and HRE15–291 [focus groups]). All survey participants provided informed consent to partake in the study prior to undertaking the first survey by clicking ‘agree’ in the survey software (online participants) or signing a written consent form (paper participants) after they had read all participant information relating to the study. Focus group participants also completed a written consent form after reading and understanding all participant information regarding the study.

Consent for publication

Focus group participants have given permission for their quotes to be used in de-identified form prior to undertaking the research.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Lindsay-Smith, G., O’Sullivan, G., Eime, R. et al. A mixed methods case study exploring the impact of membership of a multi-activity, multicentre community group on social wellbeing of older adults. BMC Geriatr 18 , 226 (2018). https://doi.org/10.1186/s12877-018-0913-1

Download citation

Received : 19 April 2018

Accepted : 10 September 2018

Published : 24 September 2018

DOI : https://doi.org/10.1186/s12877-018-0913-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Social engagement
• Group activity

BMC Geriatrics

ISSN: 1471-2318

• Submission enquiries: [email protected]
• General enquiries: [email protected]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Mixed Methods Research | Definition, Guide & Examples

Mixed Methods Research | Definition, Guide & Examples

Published on August 13, 2021 by Tegan George . Revised on June 22, 2023.

Mixed methods research combines elements of quantitative research and qualitative research in order to answer your research question . Mixed methods can help you gain a more complete picture than a standalone quantitative or qualitative study, as it integrates benefits of both methods.

Mixed methods research is often used in the behavioral, health, and social sciences, especially in multidisciplinary settings and complex situational or societal research.

• To what extent does the frequency of traffic accidents ( quantitative ) reflect cyclist perceptions of road safety ( qualitative ) in Amsterdam?
• How do student perceptions of their school environment ( qualitative ) relate to differences in test scores ( quantitative ) ?
• How do interviews about job satisfaction at Company X ( qualitative ) help explain year-over-year sales performance and other KPIs ( quantitative ) ?
• How can voter and non-voter beliefs about democracy ( qualitative ) help explain election turnout patterns ( quantitative ) in Town X?
• How do average hospital salary measurements over time (quantitative) help to explain nurse testimonials about job satisfaction (qualitative) ?

Table of contents

When to use mixed methods research, mixed methods research designs, advantages of mixed methods research, disadvantages of mixed methods research, other interesting articles, frequently asked questions.

Mixed methods research may be the right choice if your research process suggests that quantitative or qualitative data alone will not sufficiently answer your research question. There are several common reasons for using mixed methods research:

• Generalizability : Qualitative research usually has a smaller sample size , and thus is not generalizable. In mixed methods research, this comparative weakness is mitigated by the comparative strength of “large N,” externally valid quantitative research.
• Contextualization: Mixing methods allows you to put findings in context and add richer detail to your conclusions. Using qualitative data to illustrate quantitative findings can help “put meat on the bones” of your analysis.
• Credibility: Using different methods to collect data on the same subject can make your results more credible. If the qualitative and quantitative data converge, this strengthens the validity of your conclusions. This process is called triangulation .

As you formulate your research question , try to directly address how qualitative and quantitative methods will be combined in your study. If your research question can be sufficiently answered via standalone quantitative or qualitative analysis, a mixed methods approach may not be the right fit.

But mixed methods might be a good choice if you want to meaningfully integrate both of these questions in one research study.

Keep in mind that mixed methods research doesn’t just mean collecting both types of data; you need to carefully consider the relationship between the two and how you’ll integrate them into coherent conclusions.

Mixed methods can be very challenging to put into practice, and comes with the same risk of research biases as standalone studies, so it’s a less common choice than standalone qualitative or qualitative research.

Prevent plagiarism. Run a free check.

There are different types of mixed methods research designs . The differences between them relate to the aim of the research, the timing of the data collection , and the importance given to each data type.

As you design your mixed methods study, also keep in mind:

• Your research approach ( inductive vs deductive )
• Your research questions
• What kind of data is already available for you to use
• What kind of data you’re able to collect yourself.

Here are a few of the most common mixed methods designs.

Convergent parallel

In a convergent parallel design, you collect quantitative and qualitative data at the same time and analyze them separately. After both analyses are complete, compare your results to draw overall conclusions.

• On the qualitative side, you analyze cyclist complaints via the city’s database and on social media to find out which areas are perceived as dangerous and why.
• On the quantitative side, you analyze accident reports in the city’s database to find out how frequently accidents occur in different areas of the city.

In an embedded design, you collect and analyze both types of data at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.

This is a good approach to take if you have limited time or resources. You can use an embedded design to strengthen or supplement your conclusions from the primary type of research design.

Explanatory sequential

In an explanatory sequential design, your quantitative data collection and analysis occurs first, followed by qualitative data collection and analysis.

You should use this design if you think your qualitative data will explain and contextualize your quantitative findings.

Exploratory sequential

In an exploratory sequential design, qualitative data collection and analysis occurs first, followed by quantitative data collection and analysis.

You can use this design to first explore initial questions and develop hypotheses . Then you can use the quantitative data to test or confirm your qualitative findings.

“Best of both worlds” analysis

Combining the two types of data means you benefit from both the detailed, contextualized insights of qualitative data and the generalizable , externally valid insights of quantitative data. The strengths of one type of data often mitigate the weaknesses of the other.

For example, solely quantitative studies often struggle to incorporate the lived experiences of your participants, so adding qualitative data deepens and enriches your quantitative results.

Solely qualitative studies are often not very generalizable, only reflecting the experiences of your participants, so adding quantitative data can validate your qualitative findings.

Method flexibility

Mixed methods are less tied to disciplines and established research paradigms. They offer more flexibility in designing your research, allowing you to combine aspects of different types of studies to distill the most informative results.

Mixed methods research can also combine theory generation and hypothesis testing within a single study, which is unusual for standalone qualitative or quantitative studies.

Mixed methods research is very labor-intensive. Collecting, analyzing, and synthesizing two types of data into one research product takes a lot of time and effort, and often involves interdisciplinary teams of researchers rather than individuals. For this reason, mixed methods research has the potential to cost much more than standalone studies.

Differing or conflicting results

If your analysis yields conflicting results, it can be very challenging to know how to interpret them in a mixed methods study. If the quantitative and qualitative results do not agree or you are concerned you may have confounding variables , it can be unclear how to proceed.

Due to the fact that quantitative and qualitative data take two vastly different forms, it can also be difficult to find ways to systematically compare the results, putting your data at risk for bias in the interpretation stage.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Non-probability sampling
• Quantitative research
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Triangulation in research means using multiple datasets, methods, theories and/or investigators to address a research question. It’s a research strategy that can help you enhance the validity and credibility of your findings.

Triangulation is mainly used in qualitative research , but it’s also commonly applied in quantitative research . Mixed methods research always uses triangulation.

These are four of the most common mixed methods designs :

• Convergent parallel: Quantitative and qualitative data are collected at the same time and analyzed separately. After both analyses are complete, compare your results to draw overall conclusions. 
• Embedded: Quantitative and qualitative data are collected at the same time, but within a larger quantitative or qualitative design. One type of data is secondary to the other.
• Explanatory sequential: Quantitative data is collected and analyzed first, followed by qualitative data. You can use this design if you think your qualitative data will explain and contextualize your quantitative findings.
• Exploratory sequential: Qualitative data is collected and analyzed first, followed by quantitative data. You can use this design if you think the quantitative data will confirm or validate your qualitative findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, June 22). Mixed Methods Research | Definition, Guide & Examples. Scribbr. Retrieved August 19, 2024, from https://www.scribbr.com/methodology/mixed-methods-research/

Is this article helpful?

Tegan George

Other students also liked, writing strong research questions | criteria & examples, what is quantitative research | definition, uses & methods, what is qualitative research | methods & examples, what is your plagiarism score.

• Open access
• Published: 16 August 2024

Going virtual: mixed methods evaluation of online versus in-person learning in the NIH mixed methods research training program retreat

• Joseph J. Gallo 1 ,
• Sarah M. Murray 1 ,
• John W. Creswell 2 ,
• Charles Deutsch 3 &
• Timothy C. Guetterman 2  

BMC Medical Education volume  24 , Article number:  882 ( 2024 ) Cite this article

63 Accesses

Metrics details

Despite the central role of mixed methods in health research, studies evaluating online methods training in the health sciences are nonexistent. The focused goal was to evaluate online training by comparing the self-rated skills of scholars who experienced an in-person retreat to scholars in an online retreat in specific domains of mixed methods research for the health sciences from 2015–2023.

The authors administered a scholar Mixed Methods Skills Self-Assessment instrument based on an educational competency scale that included domains on: “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination” to participants of the Mixed Methods Research Training Program for the Health Sciences (MMRTP). Self-ratings on confidence on domains were compared before and after retreat participation within cohorts who attended in person ( n  = 73) or online ( n  = 57) as well as comparing across in-person to online cohorts. Responses to open-ended questions about experiences with the retreat were analyzed.

Scholars in an interactive program to improve mixed methods skills reported significantly increased confidence in ability to define or explain concepts and in ability to apply the concepts to practical problems, whether the program was attended in-person or synchronously online. Scholars in the online retreat had self-rated skill improvements as good or better than scholars who participated in person. With the possible exception of networking, scholars found the online format was associated with advantages such as accessibility and reduced burden of travel and finding childcare. No differences in difficulty of learning concepts was described.

Conclusions

Keeping in mind that the retreat is only one component of the MMRTP, this study provides evidence that mixed methods training online was associated with the same increases in self-rated skills as persons attending online and can be a key component to increasing the capacity for mixed methods research in the health sciences.

Peer Review reports

Introduction

The coronavirus pandemic accelerated interest in distance or remote learning. While the acute nature of the pandemic has abated, changes in the way people work have largely remained, with hybrid conferences and trainings more commonly implemented now than during the pre-pandemic period. Studies of health-related online teaching have focused on medical students [ 1 , 2 , 3 ], health professionals [ 4 , 5 ], and medical conferences [ 6 , 7 , 8 ] and have touted the advantages of virtual training and conferences in health education, but few studies have assessed relative growth in skills and competencies in health research methods for synchronous online vs. in-person training.

The National Institutes of Health (NIH)-funded Mixed Methods Research Training Program (MMRTP) for the Health Sciences provided training to faculty-level investigators across health disciplines from 2015–2023. The NIH is a major funder of health-related research in the United States. Its institutes span diseases and conditions (e.g., mental health, environmental health) in addition to focus areas (e.g., minority health and health disparities, nursing) and developing research capacity. Scholars in the MMRTP seek to develop skills in mixed methods research through participation in a summer retreat followed by ongoing mentorship for one year from a mixed methods expert matched to the scholar to support their development of a research proposal. Webinars leading up to the retreat include didactic sessions taught by the same faculty each year, and the retreat itself contains multiple interactive small group sessions in which each scholar presents their project and receives feedback on their grant proposal. Due to pandemic restrictions on gatherings and travel, in 2020 the MMRTP retained all components of the program but transitioned the in-person retreat to a synchronous online retreat.

The number of NIH agencies funding mixed methods research increased from 23 in 1997–2008 to 36 in 2009–2014 [ 9 ]. The usefulness of mixed methods research aligns with several Institutes’ strategic priories, including improving health equity, enhancing feasibility, acceptability, and sustainability of interventions, and addressing patient-centeredness. However, there is a tension between growing interest in mixed methods for health sciences research and a lack of training for investigators to acquire mixed methods research skills. Mixed methods research is not routinely taught in doctoral programs, institutional grant-writing programs, nor research training that academic physicians receive. The relative lack of researchers trained in mixed methods research necessitates ongoing research capacity building and mentorship [ 10 ]. Online teaching has the potential to meet growing demand for training and mentoring in mixed methods, as evidenced by the growth of online offerings by the Mixed Methods International Research Association [ 11 ]. Yet, the nature of skills and attitudes required for doing mixed methods research, such as integration of quantitative and qualitative data collection, analysis, and epistemologies, may make this type of training difficult to adapt to an online format without compromising its effectiveness.

Few studies have attempted to evaluate mixed methods training [ 12 , 13 , 14 , 15 ] and none appear to have evaluated online trainings in mixed methods research. Our goal was to evaluate our online MMRTP by comparing the self-rated skills of scholars who experienced an in-person retreat to an online retreat across specific domains. While the MMRTP retreat is only one component of the program, assessment before and after the retreat among persons who experienced the synchronous retreat online compared to in-person provides an indication of the effectiveness of online instruction in mixed methods for specific domains critical to the design of research in health services. We hypothesized that scholars who attended the retreat online would exhibit improvements in self-rated skills comparable to scholars who attended in person.

Participants

Five cohorts with a total of 73 scholars participated in the MMRTP in person (2015–2019), while four cohorts with a total of 57 scholars participated online (2020–2023). Scholars are faculty-level researchers in the health sciences in the United States. The scholars are from a variety of disciplines in the health sciences; namely, pediatrics, psychiatry, general medicine, oncology, nursing, human development, music therapy, nutrition, psychology, and social work.

The mixed methods research training program

Formal program activities include two webinars leading up to a retreat followed by ongoing mentorship support. The mixed methods content taught in webinars and the retreat is informed by a widely used textbook by Creswell and Plano Clark [ 18 ] in addition to readings on methodological topics and the practice of mixed methods. The webinars introduce mixed methods research and integration concepts, with the goal of imparting foundational knowledge and ensuring a common language. Specifically, the first webinar introduces mixed methods concepts, research designs, scientific rigor, and becoming a resource at one’s institution, while the second focuses on strategies for the integration of qualitative and quantitative research. Retreats provide an active workshop blending lectures, one-on-one meetings, and interactive faculty-led small workgroups. In addition to scholars, core program faculty who serve as investigators and mentors for the MMRTP, supplemented with consultants and former scholars, lead the retreat. The retreat has covered the state-of-the-art topics within the context of mixed methods research: rationale for use of mixed methods, procedural diagrams, study aims, use of theory, integration strategies, sampling strategies, implementation science, randomized trials, ethics, manuscript and proposal writing, and becoming a resource at one’s home institution. In addition to lectures, the retreat includes multiple interactive small group sessions in which each scholar presents their project and receives feedback on their grant proposal and is expected to make revisions based on feedback and lectures.

Scholars are matched for one year with a mentor based on the Scholar’s needs, career level, and area of health research from a national list of affiliated experienced mixed methods investigators with demonstrated success in obtaining independent funding for research related to the health sciences and a track record and commitment to mentoring. The purpose of this arrangement is to provide different perspectives on mixed methods design while also providing specific feedback on the scholar's research proposal, reviewing new ideas, and together developing a strategy and timeline for submission.

From 2015–2019 (in-person cohorts) the retreat was held over 3 days at the Johns Hopkins University Bloomberg School of Public Health (in 2016 Harvard Catalyst, the Harvard Clinical and Translational Science Center, hosted the retreat at Harvard Medical School). Due to pandemic restrictions, from 2020–2023 the retreat activities were conducted via Zoom with the same number of lecture sessions (over 3 days in 2020 and 4 days thereafter). We made adaptations for the online retreat based on continuous feedback from attendees. We had to rapidly transition to online in 2020 with the same structure as in person, but feedback from scholars led us to extend the retreat to 4 days online from 2021–2023. The extra day allowed for more breaks from Zoom sessions with time for scholars to consider feedback from small groups and to have one-on-one meetings with mentors. Discussion during interactive presentations was encouraged and facilitated by using breakout rooms at breaks mid-presentation. Online resources were available to participants through CoursePlus, the teaching and learning platform used for courses at the Johns Hopkins Bloomberg School of Public Health, hosting publications, presentation materials, recordings of lectures, sharing proposals, email, and discussion boards that scholars have access to before, during, and after the retreat.

Measurement strategy

Before and after the retreat in each year, we distributed a self-administered scholar Mixed Methods Skills Self-Assessment instrument (Supplement 1) to all participating scholars [ 15 ]; we have reported results from this pre-post assessment for the first two cohorts [ 14 ]. The Mixed Methods Skills Self-Assessment instrument has been previously used and has established reliability for the total items (α = 0.95) and evidence of criterion-related validity between experiences and ability ratings [ 15 ]. In each year, the pre-assessment is completed upon entry to the program, approximately four months prior to the retreat, and the post-assessment is administered two weeks after the retreat. The instrument consists of three sections: 1) professional experiences with mixed methods, including background, software, and resource familiarity; 2) a quantitative, qualitative, and mixed methods skills self-assessment; and 3) open-ended questions focused on learning goals for the MMRTP. The skills assessment contains items for each of the following domains: “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination.” Each skill was assessed via three items drawn from an educational competency ratings scale that ask scholars to rate: [ 16 ] “My ability to define/explain,” “My ability to apply to practical problems,” and “Extent to which I need to improve my skill.” Response options were on a five-point Likert-type scale that ranged from “Not at all” (coded ‘1’) to “To a great extent” (coded ‘5’), including a mid-point [ 17 ]. We took the mean of the scholar’s item ratings over all component items within each domain (namely, “research questions,” “design/approach,” “sampling,” “analysis,” and “dissemination”).

Open-ended questions

The baseline survey included two open-ended prompts: 1) What skills and goals are most important to you?, and 2) What would you like to learn? The post-assessment survey also included two additional open-ended questions about the retreat: 1) What aspects of the retreat were helpful?, and 2) What would you like to change about the retreat? In addition, for the online cohorts (2020–2023), we wanted to understand reactions to the online training and added three questions for this purpose: (1) In general, what did you think of the online format for the MMRTP retreat?, 2) What mixed methods concepts are easier or harder to learn virtually?, and 3) What do you think was missing from having the retreat online rather than in person?

Data analysis

Our evaluation employed a convergent mixed methods design [ 18 ], integrating an analysis of ratings pre- and post-retreat with analysis of open-ended responses provided by scholars after the retreat. Our quantitative analysis proceeded in 3 steps. First, we analyzed item-by-item baseline ratings of the extent to which scholars thought they “need to improve skills,” stratified into two groups (5 cohorts who attended in-person and 4 cohorts who attended online). The purpose of comparing the two groups at baseline on learning needs was to assess how similar the scholars in the in-person or online groups were in self-assessment of learning needs before attending the program. Second, to examine the change in scholar ratings of ability to “define or explain a concept” and in their ability to “apply to practical problems,” from before to after the retreat, we conducted paired t-tests. The goal was to compare the ratings before and after the retreat among scholars who attended the program in person to scholars who attended online. Third, we compared post-retreat ratings among in-person cohorts to online cohorts to gauge the effectiveness of the online training. We set statistical significance at α  < 0.05 as a guide to inference. We calculated Cohen’s d as a guide to the magnitude of differences [ 19 ]. SPSS Version 28 was employed for all analyses.

We analyzed qualitative data using a thematic analysis approach that consisted of reviewing all open-ended responses, conducting open coding based on the data, developing and refining a codebook, and identifying major themes [ 20 ]. We then compared the qualitative results for the in-person versus online cohorts to understand any thematic differences concerning retreat experiences and reactions.

Background and experiences of scholars

Scholars in the in-person ( n  = 59, 81%) and online ( n  = 52, 91%) cohorts reported their primary training was quantitative rather than qualitative or mixed methods, and scholars across cohorts commonly reported at least some exposure to mixed methods research (Table  1 ). However, most scholars did not have previous mixed methods training with 17 (23%) and 16 (28%) of the in-person and online cohorts, respectively, having previously completed a mixed methods course. While experiences were similar across in-person vs. online cohorts, there were two areas in which the scholars reported a statistically significant difference: a larger portion of the online cohorts reported writing a mixed methods application that received funding ( n  = 35, 48% in person; n  = 46, 81% online), and a smaller proportion of the online cohorts had given a local or institutional mixed methods presentation ( n  = 32, 44% in person; n  = 15, 26% online).

Self-identified need to improve skills in mixed methods

At baseline, scholars rated the extent to which they needed to improve specific mixed methods skills (Table  2 ). Overall, scholars endorsed a strong need to improve all mixed methods skills. The ratings between the in-person and online cohorts were not statistically significant for any item.

Change in self-ratings of skills after the retreat

Within cohorts.

For all domains, the differences in pre-post assessment scores were statistically significant for both the in-person and online cohorts in ability to define or explain concepts and to apply concepts to practical problems (left side of Table  3 ). In other words, on average scholars improved in both in-person and online cohorts.

Across cohorts

Online cohorts had significantly better self-ratings after the retreat than did in-person cohorts in ability to define or explain concepts and to apply concepts to practical problems (in sampling, data collection, analysis, and dissemination) but no significant differences in research questions and design / approach (rightmost column of Table  3 ).

Scholar reflections about online and in-person retreats

Goals of training.

In comparing in-person to online cohorts, discussions of the skills that scholars wanted to improve had no discernable differences. Scholars mentioned wanting to develop skills in the foundations of mixed methods research, how to write competitive proposals for funding, the use of the terminology of mixed methods research, and integrative analysis. In addition, some scholars expressed wanting to become a resource at their own institutions and providing training and mentoring to others.

Small group sessions

Scholars consistently reported appreciating being able to talk through their project and gaining feedback from experts in small group sessions. Some scholars expressed a preference for afternoon small group sessions, “The small group sessions felt the most helpful, but only because we can apply what we were learning from the morning lecture sessions” (online cohort 9). How participants discussed the benefits of the small group sessions or how they used the sessions did not depend on whether they had experienced the session in person or online.

Online participants described a tradeoff between the accessibility of a virtual retreat versus advantages of in-person training. One participant explained, “I liked the online format, as I do not have reliable childcare” (online cohort 8). Many of the scholars felt that there was an aspect of networking missing when the retreat was held fully online. As one scholar described, when learning online they, “miss getting to know the other fellows and forming lasting connections” (online cohort 9). However, an equal number of others reported that having a virtual retreat meant less hassle; for instance, they were able to join from their preferred location and did not have to travel. Some individuals specifically described the tradeoff of fewer networking opportunities for ease of attendance. One scholar wrote, being online “certainly loses some of the perks of in person connection building but made it equitable to attend” (online cohort 8).

Learning online

No clear difference in ease of learning concepts was described. A scholar explained: “Learning most concepts is essentially the same virtually versus in person” (online cohort 8). However, scholars described some concepts as easier to learn in one modality versus the other, for example, simpler concepts being more suited to learning virtually while complex concepts were better suited to in-person learning. There was notable variation though in the topics which scholars considered to be simple versus complex. For instance, one scholar noted that “I suppose developing the joint displays were a bit tougher virtually since you were not literally elbow to elbow” (online cohort 7) while another explained, “joint displays lend themselves to the zoom format” (online cohort 8).

Integrating survey responses and scholar reflections

In-person and online cohorts were comparable in professional experiences and ratings of the need to improve skills before attending the retreat, sharpening the focus on differences in self-rated skills associated with attendance online compared to in person. If anything, online attendees rated skills as good or better than in-person attendees. Open-ended questions revealed that, for the most part, scholar reflections on learning were similar across in-person and online cohorts. Whether learning the concept of “mixed methods integration” was more difficult online was a source of disagreement. Online attendance was associated with numerous advantages, and small group sessions were valued, regardless of format. Taken together, the evidence from nine cohorts shows that the online retreat was acceptable and as effective in improving self-rated skills as meeting in person.

Mixed methods have become indispensable to health services research from intervention development and testing [ 21 ] to implementation science [ 22 , 23 , 24 ]. We found that scholars participating in an interactive program to improve mixed methods skills reported significantly increased confidence in their ability to define or explain concepts and in their ability to apply the concepts to practical problems, whether the program was attended in-person or synchronously online. Scholars who participated in the online retreat had self-rated skill improvements as good or better than scholars who participated in person, and these improvements were relatively large as indicated by the Cohen’s d estimates. The online retreat appeared to be effective in increasing confidence in the use of mixed methods research in the health sciences and was acceptable to scholars. Our study deserves attention because the national need is so great for investigators with training in mixed methods to address complex behavioral health problems, community- and patient-centered research, and implementation research. No program has been evaluated as we have done here.

Aside from having written a funded mixed methods proposal, the online compared to earlier in person cohorts were comparable in experiences and need to improve specific skills. Within each cohort, scholars reported significant gains in self-rated skills on their ability to “define or explain” a concept and on their ability to “apply to practical problems” in domains essential to mixed methods research. However, consistent with our hypothesis that online training would be as effective as in person we found that online scholars reported better improvement in self-ratings in ability to define or explain concepts and to apply concepts to practical problems in sampling, data collection, analysis, and dissemination but no significant differences in research questions and design / approach. Better ratings in online cohorts could reflect differences in experience with mixed methods, secular changes in knowledge and availability of resources in mixed methods, and maturation of the program facilitated by continued modifications based on feedback from scholars and participating faculty [ 13 , 14 , 15 ].

Ratings related to the “analysis” domain, which includes the central concept of mixed methods integration, deserve notice since scholars rated this skill well below other domains at baseline. While both in-person and online cohorts improved after the retreat, and online cohorts improved substantially more than in-person cohorts, ratings for analysis after the retreat remained lower than for other domains. Scholars consistently have mentioned integration as a difficult concept, and our analysis here is limited to the retreat alone. Continued mentoring one year after the retreat and work on their proposal is built in to the MMRTP to enhance understanding of integration.

Several reviews point out the advantages of online training including savings in time, money, and greenhouse emissions [ 1 , 7 , 8 ]. Online conferences may increase the reach of training to international audiences, improve the diversity of speakers and attendees, facilitate attendance of persons with disabilities, and ease the burden of finding childcare [ 1 , 8 , 25 ]. Online training in health also appears to be effective [ 2 , 4 , 5 , 25 ], though studies are limited because often no skills were evaluated, no comparison groups were used, the response rate was low, or the sample size was small [ 1 , 6 ]. With the possible exception of networking, scholars found the online format was associated with advantages, including saving travel, maintaining work-family balance, and learning effectively. As scholars did discuss perceived increase in difficulty networking, deliberate effort needs to be directed at enhancing collaborations and mentorship [ 8 ]. The MMRTP was designed with components to facilitate networking during and beyond the retreat (e.g., small group sessions, one-on-one meetings, working with a consultant on a specific proposal).

Limitations of our study should be considered. First, the retreat was only one of several components of a mentoring program for faculty in the health sciences. Second, in-person and online cohorts represent different time periods spanning 9 years during which mixed methods applications to NIH and other funders have been increasing [ 9 ]. Third, the pre- and post-evaluations of ability to explain or define concepts, or to apply the concepts to practical problems, were based on self-report. Nevertheless, the pre-post retreat survey on self-rated skills uses a skills self-assessment form we developed [ 15 ], drawing from educational theory related to the epistemology of knowledge [ 26 , 27 ].

Despite the central role of mixed methods in health research, studies evaluating online methods training in the health sciences are nonexistent. Our study provides evidence that mixed methods training online was associated with the same increases in self-rated skills as persons attending online and can be a key component to increasing the capacity for mixed methods research in the health sciences.

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Mixed Methods Research Training Program

Wilcha RJ. Effectiveness of Virtual Medical Teaching During the COVID-19 Crisis: Systematic Review. JMIR Med Educ. 2020;6(2):e20963.

Article   Google Scholar  

Pei L, Wu H. Does online learning work better than offline learning in undergraduate medical education? A systematic review and meta-analysis. Medical Education Online. 2019;24(1) https://doi.org/10.1080/10872981.2019.1666538

Barche A, Nayak V, Pandey A, Bhandarkar A, Nayak K. Student perceptions towards online learning in medical education during the COVID-19 pandemic: a mixed-methods study. F1000Res. 2022;11:979. https://doi.org/10.12688/f1000research.123582.1 .

Ebner C, Gegenfurtner A. Learning and Satisfaction in Webinar, Online, and Face-to-Face Instruction: A Meta-Analysis. Frontiers in Education. 2019;4(92) https://doi.org/10.3389/feduc.2019.00092

Randazzo M, Preifer R, Khamis-Dakwar R. Project-Based Learning and Traditional Online Teaching of Research Methods During COVID-19: An Investigation of Research Self-Efficacy and Student Satisfaction. Frontiers in Education. 2021;6(662850) https://doi.org/10.3389/feduc.2021.662850

Chan A, Cao A, Kim L, et al. Comparison of perceived educational value of an in-person versus virtual medical conference. Can Med Educ J. 2021;12(4):65–9. https://doi.org/10.36834/cmej.71975 .

Rubinger L, Gazendam A, Ekhtiari S, et al. Maximizing virtual meetings and conferences: a review of best practices. Int Orthop. 2020;44(8):1461–6. https://doi.org/10.1007/s00264-020-04615-9 .

Sarabipour S. Virtual conferences raise standards for accessibility and interactions. Elife. Nov 4 2020;9 https://doi.org/10.7554/eLife.62668

Coyle CE, Schulman-Green D, Feder S, et al. Federal funding for mixed methods research in the health sciences in the United States: Recent trends. J Mixed Methods Res. 2018;12(3):1–20.

Poth C, Munce SEP. Commentary – preparing today’s researchers for a yet unknown tomorrow: promising practices for a synergistic and sustainable mentoring approach to mixed methods research learning. Int J Multiple Res Approaches. 2020;12(1):56–64.

Creswell JW. Reflections on the MMIRA The Future of Mixed Methods Task Force Report. J Mixed Methods Res. 2016;10(3):215–9. https://doi.org/10.1177/1558689816650298 .

Hou S. A Mixed Methods Process Evaluation of an Integrated Course Design on Teaching Mixed Methods Research. Int J Sch Teach Learn. 2021;15(2):Article 8. https://doi.org/10.20429/ijsotl.2021.150208 .

Guetterman TC, Creswell J, Deutsch C, Gallo JJ. Process Evaluation of a Retreat for Scholars in the First Cohort: The NIH Mixed Methods Research Training Program for the Health Sciences. J Mix Methods Res. 2019;13(1):52–68. https://doi.org/10.1177/1558689816674564 .

Guetterman T, Creswell JW, Deutsch C, Gallo JJ. Skills Development and Academic Productivity of Scholars in the NIH Mixed Methods Research Training Program for the Health Sciences (invited publication). Int J Multiple Res Approach. 2018;10(1):1–17.

Guetterman T, Creswell JW, Wittink MN, et al. Development of a Self-Rated Mixed Methods Skills Assessment: The NIH Mixed Methods Research Training Program for the Health Sciences. J Contin Educ Health Prof. 2017;37(2):76–82.

Harnisch D, Shope RJ. Developing technology competencies to enhance assessment literate teachers. AACE; 2007:3053–3055.

DeVellis RF. Scale development: Theory and applications. 3rd ed. Sage; 2012.

Creswell JW, Plano Clark VL. Designing and Conducting Mixed Methods Research. 3rd ed. Sage Publications; 2017.

Cohen J. Statistical power analysis for the behavioral sciences. 3rd ed. Academic Press; 1988.

Boeije H. A purposeful approach to the constant comparative method in the analysis of qualitative interviews. Qual Quant. 2002;36:391–409.

Aschbrenner KA, Kruse G, Gallo JJ, Plano Clark VL. Applying mixed methods to pilot feasibility studies to inform intervention trials. Pilot Feasibility Stud. 2022;8(1):217–24. https://doi.org/10.1186/s40814-022-01178-x .

Palinkas LA. Qualitative and mixed methods in mental health services and implementation research. J Clin Child Adolesc Psychol. 2014;43(6):851–61.

Albright K, Gechter K, Kempe A. Importance of mixed methods in pragmatic trials and dissemination and implementation research. Acad Pediatr Sep-Oct. 2013;13(5):400–7. https://doi.org/10.1016/j.acap.2013.06.010 .

Palinkas L, Aarons G, Horwitz S, Chamberlain P, Hurlburt M, Landsverk J. Mixed methods designs in implementation research. Adm Policy Ment Health. 2011;38:44–53.

Ni AY. Comparing the Effectiveness of Classroom and Online Learning: Teaching Research Methods. J Public Affairs Educ. 2013;19(2):199–215. https://doi.org/10.1080/15236803.2013.12001730 .

Harnisch D, Shope RJ. Developing technology competencies to enhance assessment literate teachers. presented at: Society for Information Technology & Teacher Education International Conference; March 26, 2007 2007; San Antonio, Texas.

Guetterman TC. What distinguishes a novice from an expert mixed methods researcher? Qual Quantity. 2017;51:377–98.

Download references

Acknowledgements

The Mixed Methods Research Training Program is supported by the Office of Behavioral and Social Sciences Research under Grant R25MH104660. Participating institutes are the National Institute of Mental Health, National Heart, Lung, and Blood Institute, National Institute of Nursing Research, and the National Institute on Aging.

Author information

Authors and affiliations.

Johns Hopkins University, Baltimore, MD, USA

Joseph J. Gallo & Sarah M. Murray

University of Michigan, Ann Arbor, MI, USA

John W. Creswell & Timothy C. Guetterman

Harvard University, Boston, MA, USA

Charles Deutsch

You can also search for this author in PubMed   Google Scholar

Contributions

All authors conceptualized the design of this study. TG analyzed the scholar data in evaluation of the program. TG and JG interpreted results and were major contributors in writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Timothy C. Guetterman .

Ethics declarations

Ethics approval and consent to participate.

The program was reviewed by the Johns Hopkins Institutional Review Board and was deemed exempt as educational research under United States 45 CFR 46.101(b), Category (2). Data were collected through an anonymous survey. Consent to participate was waived.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Gallo, J.J., Murray, S.M., Creswell, J.W. et al. Going virtual: mixed methods evaluation of online versus in-person learning in the NIH mixed methods research training program retreat. BMC Med Educ 24 , 882 (2024). https://doi.org/10.1186/s12909-024-05877-2

Download citation

Received : 15 January 2024

Accepted : 08 August 2024

Published : 16 August 2024

DOI : https://doi.org/10.1186/s12909-024-05877-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research training
• Mixed methods research
• Research capacity building
• Online education
• Teaching methods

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Mixed methods research

Affiliation.

• 1 University of Wollongong, Australia.
• PMID: 25850508
• DOI: 10.7748/ns.29.32.41.e8858

Mixed methods research involves the use of qualitative and quantitative data in a single research project. It represents an alternative methodological approach, combining qualitative and quantitative research approaches, which enables nurse researchers to explore complex phenomena in detail. This article provides a practical overview of mixed methods research and its application in nursing, to guide the novice researcher considering a mixed methods research project.

Keywords: Mixed methods research; multi-method research; nursing research; qualitative research; quantitative research; research design; research methodology; rigour.

PubMed Disclaimer

Similar articles

• Clarifying case study research: examples from practice. Casey D, Houghton C. Casey D, et al. Nurse Res. 2010;17(3):41-51. doi: 10.7748/nr2010.04.17.3.41.c7745. Nurse Res. 2010. PMID: 20450088
• Rigour in phenomenological research: reflections of a novice nurse researcher. Pereira HR. Pereira HR. Nurse Res. 2012;19(3):16-9. doi: 10.7748/nr2012.04.19.3.16.c9054. Nurse Res. 2012. PMID: 22662373
• Differentiating between descriptive and interpretive phenomenological research approaches. Matua GA, Van Der Wal DM. Matua GA, et al. Nurse Res. 2015 Jul;22(6):22-7. doi: 10.7748/nr.22.6.22.e1344. Nurse Res. 2015. PMID: 26168810 Review.
• Stages in the research process. Gelling L. Gelling L. Nurs Stand. 2015 Mar 4;29(27):44-9. doi: 10.7748/ns.29.27.44.e8745. Nurs Stand. 2015. PMID: 25736674
• Searching for rigour in the reporting of mixed methods population health research: a methodological review. Brown KM, Elliott SJ, Leatherdale ST, Robertson-Wilson J. Brown KM, et al. Health Educ Res. 2015 Dec;30(6):811-39. doi: 10.1093/her/cyv046. Epub 2015 Oct 21. Health Educ Res. 2015. PMID: 26491072 Review.
• Perioperative mental health intervention for depression and anxiety symptoms in older adults study protocol: design and methods for three linked randomised controlled trials. Holzer KJ, Bartosiak KA, Calfee RP, Hammill CW, Haroutounian S, Kozower BD, Cordner TA, Lenard EM, Freedland KE, Tellor Pennington BR, Wolfe RC, Miller JP, Politi MC, Zhang Y, Yingling MD, Baumann AA, Kannampallil T, Schweiger JA, McKinnon SL, Avidan MS, Lenze EJ, Abraham J. Holzer KJ, et al. BMJ Open. 2024 Apr 3;14(4):e082656. doi: 10.1136/bmjopen-2023-082656. BMJ Open. 2024. PMID: 38569683 Free PMC article.
• Sexual and reproductive health (SRH) of women with mental illness (WMI) - An integrative mixed-method study. Vijayalakshmi S, Rajagopal K, Govindan R, Ganjekar S, Chacko LK, Prathyusha PV. Vijayalakshmi S, et al. Indian J Psychiatry. 2024 Feb;66(2):171-178. doi: 10.4103/indianjpsychiatry.indianjpsychiatry_524_23. Epub 2024 Feb 12. Indian J Psychiatry. 2024. PMID: 38523762 Free PMC article.
• Hospital pharmacists' mental health during home isolation in the post-pandemic era of COVID-19: influencing factors, coping strategies, and the mediating effect of resilience. Yin Z, Wang X, Lu X, Fu H. Yin Z, et al. Front Public Health. 2024 Jan 31;12:1268638. doi: 10.3389/fpubh.2024.1268638. eCollection 2024. Front Public Health. 2024. PMID: 38356948 Free PMC article.
• Female genital self-image and body appreciation in Turkish women: a mixed methods study. Koçak V, Aksoy YE. Koçak V, et al. Arch Womens Ment Health. 2024 Apr;27(2):219-232. doi: 10.1007/s00737-023-01397-y. Epub 2023 Nov 25. Arch Womens Ment Health. 2024. PMID: 38006426
• The Effects of Quality Assurance System Implementation on Work Well-Being and Patient Safety: Protocol for a Mixed Methods Study. Vuohijoki A, Huusko M, Ristolainen L, Hakasaari P, Kautiainen H, Leppilahti J, Kivivuori SM, Hurri H. Vuohijoki A, et al. JMIR Res Protoc. 2023 Nov 23;12:e45200. doi: 10.2196/45200. JMIR Res Protoc. 2023. PMID: 37995119 Free PMC article.

Publication types

• Search in MeSH
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

• Study Protocol
• Open access
• Published: 19 August 2024

Evaluating the impact of the global evidence, local adaptation (GELA) project for enhancing evidence-informed guideline recommendations for newborn and young child health in three African countries: a mixed-methods protocol

• Tamara Kredo   ORCID: orcid.org/0000-0001-7115-9535 1 , 2 , 3 , 4 ,
• Emmanuel Effa 5 ,
• Nyanyiwe Mbeye 6 ,
• Denny Mabetha 1 ,
• Bey-Marrié Schmidt 1 , 7 ,
• Anke Rohwer 2 ,
• Michael McCaul 2 ,
• Idriss Ibrahim Kallon 2 ,
• Susan Munabi-Babigumira 8 ,
• Claire Glenton 8 ,
• Taryn Young 2 ,
• Simon Lewin 1 , 9 ,
• Per Olav Vandvik 10 , 11 &
• Sara Cooper 2 , 4 , 12  

Health Research Policy and Systems volume  22 , Article number:  114 ( 2024 ) Cite this article

Metrics details

Poverty-related diseases (PRD) remain amongst the leading causes of death in children under-5 years in sub-Saharan Africa (SSA). Clinical practice guidelines (CPGs) based on the best available evidence are key to strengthening health systems and helping to enhance equitable health access for children under five. However, the CPG development process is complex and resource-intensive, with substantial scope for improving the process in SSA, which is the goal of the Global Evidence, Local Adaptation (GELA) project. The impact of research on PRD will be maximized through enhancing researchers and decision makers’ capacity to use global research to develop locally relevant CPGs in the field of newborn and child health. The project will be implemented in three SSA countries, Malawi, South Africa and Nigeria, over a 3-year period. This research protocol is for the monitoring and evaluation work package of the project. The aim of this work package is to monitor the various GELA project activities and evaluate the influence these may have on evidence-informed decision-making and guideline adaptation capacities and processes. The specific project activities we will monitor include (1) our ongoing engagement with local stakeholders, (2) their capacity needs and development, (3) their understanding and use of evidence from reviews of qualitative research and, (4) their overall views and experiences of the project.

We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change. A series of interconnected qualitative and quantitative data collections methods will be used, including knowledge translation tracking sheets and case studies, capacity assessment online surveys, user testing and in-depth interviews, and non-participant observations of project activities. Participants will comprise of project staff, members of the CPG panels and steering committees in Malawi, South Africa and Nigeria, as well as other local stakeholders in these three African countries.

Ongoing monitoring and evaluation will help ensure the relationship between researchers and stakeholders is supported from the project start. This can facilitate achievement of common goals and enable researchers in South Africa, Malawi and Nigeria to make adjustments to project activities to maximize stakeholder engagement and research utilization. Ethical approval has been provided by South African Medical Research Council Human Research Ethics Committee (EC015-7/2022); The College of Medicine Research and Ethics Committee, Malawi (P.07/22/3687); National Health Research Ethics Committee of Nigeria (01/01/2007).

Peer Review reports

Sub-Saharan Africa (SSA) has the highest under-five mortality rate in the world [ 1 ]. Although the global under-five mortality rate declined from 76 to 38 per 1000 live births between 2000 and 2019, more than half of the deaths in children and youth in 2019 were among children under 5 years, approximately 5.2 million deaths [ 1 ]. Poverty-related diseases including pneumonia, diarrhoea and malaria remain amongst the leading causes of death in children under-5 years [ 2 ].Thus, despite progress in the health of young children globally, most countries in SSA fall below the average gains and do not meet maternal and child health targets set by the United Nations Sustainable Development Goal 3 to ‘ensure healthy lives and promote wellbeing’ (1). As of December 2021, under-five mortality rates were reported as 113.8, 38.6 and 32.2 per 1000 live births for Nigeria, Malawi and South Africa, respectively [ 3 ]. Factors accounting for regional disparities in child mortality rates include poverty, socioeconomic inequities, poor health systems and poor nutrition, with coronavirus disease 2019 (COVID-19) adding substantially to the burden [ 4 ].

Addressing healthcare issues such as these requires an evidence-informed approach, where intervention design and implementation are based on the best available evidence, to ensure that scarce resources are used effectively and efficiently, avoid harm, maximize good and improve healthcare delivery and outcomes [ 5 , 6 , 7 ]. Evidence-informed practices have been growing in SSA [ 6 ], and evidence ecosystems are becoming stronger. The evidence ecosystem reflects the formal and informal linkages and interactions between different actors (and their capacities and resources) involved in the production, translation and use of evidence [ 6 , 8 , 9 ]. Guidance that can be developed through this ecosystem includes evidence-based health technology assessments (HTA) and clinical practice guidelines (CPGs). CPGs include recommendations that are actionable statements that are informed by systematic reviews of evidence, and an assessment of the benefits and harms of alternative care options and are intended to optimize patient care [ 10 ]. They can help bridge the gap between research evidence and practice and are recognized as important quality-improvement tools that aim to standardize care, inform funding decisions and improve access to care, among others.

CPG method advancements, challenges and research gaps

Over the past decade, internationally and in SSA, there has been a rapid growth of CPGs developed for a range of conditions [ 11 ]. In particular, rapid evidence syntheses and guideline development methods has advanced in response to urgent evidence needs, especially during COVID [ 12 , 13 ]. For example, WHO has developed guidelines for all key infectious conditions that cause most deaths. This development has been accompanied by a growing volume of research evidence around CPGs, including the processes for their rapid development, adaptation, contextualization, implementation and evaluation, and further spurred on by COVID. For example, global knowledge leaders, such as the WHO and the GRADE Working Group, have set standards for CPG development, outlining the steps of what is known as ‘de novo’ (from scratch) CPG development [ 14 ]. Another global group, the Guidelines International Network (G-I-N), is a network dedicated to leading, strengthening and supporting collaboration in CPG development, adaptation and implementation. They have published minimum standards and the G-I-N McMaster guideline checklist, which contains a comprehensive list of topics and items outlining the practical steps to consider for developing CPGs [ 15 ].

As CPG standards have evolved, however, so has the complexity of development and adaptation. In the context of poorer settings, such as sub-Saharan Africa (SSA), CPG development is prohibitively human and finance resource intensive. It requires scarce skills, even in the growing evidence-based healthcare (EBHC) community, and financial investments by government where resources are often directed to healthcare services, rather than policymaking processes. Against this backdrop, several studies have found that CPGs in the region often perform poorly on reporting on their rigour of development and editorial independence [ 16 , 17 , 18 ]. Other, more resource-efficient methods for guideline development in SSA are, therefore, essential and urgently needed. Moreover, investment in the overall management of the process is needed, including convening the guideline group and moving stepwise through a rigorous process.

Approaches for and challenges of guideline adaptation

There is also increased international recognition of the value of taking guidelines developed in one country and applying them to other countries. This can avoid duplication of effort and research waste in de novo guideline development, when useful guidelines may exist elsewhere [ 12 , 19 ]. Against this backdrop, several adaptation methods are emerging for contextualization of recommendations to country needs (e.g. ADAPTE, adolopment and SNAP-it, amongst others) [ 19 , 20 , 21 ]. For example, WHO is developing strategies for adapting and implementing their CPGs at country level. One example is the WHO Antenatal Care Recommendations Adaptation Toolkit lead by the Department of Sexual and Reproductive Health and Research [ 22 ]. Their approach is pragmatic and transparent. Another approach is so-called ‘adolopment’, a GRADE method, in which the original guideline evidence is used, either adopted or adapted, considering contextual evidence such as costs and feasibility and local values [ 20 ]. Adolopment involves convening a guideline panel, reviewing available evidence and local contextual evidence and weighing up the panel’s judgements to make recommendations that are fit for purpose [ 20 ].

Despite these advances in CPG adaptation methods, many countries and professional associations in sub-Saharan Africa still use expert opinion-based approaches or proceed to prepare their own systematic reviews and guidelines, ultimately perpetuating resource wastage and duplication of efforts [ 23 ]. Moreover, when countries do adapt and contextualize other countries’ guidelines, there is frequently a lack of transparency and reporting on changes, without clarity on why or by whom. This in turn casts doubts on the recommendation’s credibility. For example, guidelines for child health in sub-Saharan Africa are usually derived from the WHO and UNICEF. However, adaptation of such guidelines and recommendations to national contexts is not well described [ 24 ]. Transparency in guideline adaptation is critical for creating trustworthy, context-sensitive recommendations. What guideline adaptation methods work best and how these can be transparently implemented in the context of lower resource settings, remain key research questions. Therefore, despite the emergence of several guideline adaptation approaches, we need to explore and understand how best to adapt recommendations from one context to another [ 25 ].

Qualitative evidence to inform guideline panels decisions

Another major advancement within guideline research has been growing recognition of the potential contribution of qualitative research evidence [ 26 , 27 ]. Traditionally, guidelines have been informed by systematic reviews of the effectiveness of specific interventions [ 14 ]. Such reviews provide robust evidence about which interventions ‘work’. However, there is appreciation that evidence regarding the potential effectiveness of an intervention is not sufficient for making recommendations or decisions. Policymakers also need to consider other issues, including how different stakeholders’ value different outcomes, the intervention’s acceptability to those affected by it and the feasibility of implementing the intervention [ 28 , 29 , 30 ]. Evidence from qualitative research is particularly well suited to exploring factors that influence an intervention’s acceptability and feasibility [ 31 , 32 ]. The use of qualitative research to inform recommendations by guidelines has become easier in recent years as systematic reviews of qualitative studies have become more common, and the methods for these reviews are now well developed [ 33 ]. The first WHO guideline to systematically incorporate reviews of qualitative studies was published in 2012 in the field of task-shifting for maternal and child health [ 31 ]. The inclusion of this qualitative evidence helped shape the panel’s recommendations [ 32 ], and this approach is now included in the WHO Handbook for Guideline Development and has been applied in many other WHO CPGs [ 34 , 35 ].

However, a key challenge in using findings from systematic reviews of qualitative evidence is communicating often complex findings to users such as guideline panel members to facilitate effective knowledge translation. While there is now considerable research on communicating findings from reviews of intervention effectiveness [ 36 ], there is limited experience on the usefulness of different options for packaging and presenting findings from systematic reviews of qualitative evidence to CPG panels. To make best use of this evidence, we need presentation formats that are accessible to users who may be unfamiliar with qualitative methods, are concise and simple while retaining sufficient detail to inform decisions and clearly present ‘confidence in the evidence from systematic reviews of qualitative evidence’ (GRADE-CERQual) assessments of how much confidence users should place in each finding [ 37 ]. In addition, we need to understand how qualitative evidence included in global guidelines, such as those produced by WHO, is interpreted and used in country-level guideline adaptation processes.

Communicating clinical practice guidelines to end-users

A final key guideline method advancement has been around the development of multi-layered and digitally structured communication formats for end users [ 38 , 39 ]. Guidelines are not an end in themselves. Recommendations may lack impact if not adequately communicated and disseminated to those who need to implement them, namely healthcare providers, managers and the public. Indeed, in a South African study of primary care guideline national policymakers, subnational health managers and healthcare providers agreed that dissemination is a particular gap [ 40 ]. While guidelines typically are produced as static documents (e.g. PDF formats), information technology is needed to enhance dissemination. The MAGIC authoring and publication Platform (MAGICapp/) was developed for this purpose ( https://magicevidence.org/magicapp/ ). MAGICapp is a web-based tool that enables evidence synthesizers and guideline organizations to create, publish and dynamically update trustworthy and digitally structured evidence summaries, guidelines and decision aids in user-friendly formats on all devices. Such digital multi-layered formats allow different users to rapidly find recommendations, while having the supporting evidence for them one click away [ 41 ]. MAGICapp, used by WHO, NICE and professional societies across the world, holds potential to enhance the impact of evidence-informed guideline recommendations in practice, in an enhanced evidence ecosystem [ 9 ]. However, the usability of the MAGICapp in sub-Saharan Africa, based on local user preferences for different communication formats, are key research questions.

Against this backdrop, the Global Evidence, Local Adaptation (GELA) project will maximize the impact of research on poverty-related diseases through enhancing researchers and decision makers’ capacity to use global research to develop locally relevant guidelines for newborn and child health in Malawi, Nigeria and South Africa. These guidelines will build on and add value to the large-scale programme of child health guideline development from agencies such as the WHO, to support adaptation and implementation led by national ministries in collaboration with WHO Afro regional office.

Brief overview of the GELA project aim, objectives and approach

The overarching aim of GELA is to bridge the gap between current processes and global advances in evidence-informed decision-making and guideline development, adaptation and dissemination by building skills and sharing resources in ways that can be sustained beyond the project period. The project has seven linked and related work packages (WPs) to support delivery of the planned project deliverables. Table 1 provides a brief summary of the activities of each WP. This protocol outlines our approach for the monitoring and overall evaluation of the project activities and impact (WP 6).

The project will be implemented in three SSA countries: Malawi, South Africa and Nigeria over a 3-year period. The project adopts a multi-faceted multidisciplinary research and capacity strengthening programme using primary and secondary research, guideline adaptation methodology and digital platforms to support authoring delivery and dynamic adaptation. These processes will offer bespoke capacity strengthening opportunities for policy makers, researchers and civil society. Throughout the project, we plan for innovations in the tools we use, accompanied by comprehensive evaluation of all aspects of the research, research uptake into policy and capacity strengthening.

This current proposal is for WP6: monitoring and evaluation

Ongoing monitoring and evaluation of project processes and activities will help facilitate ongoing engagement between researchers and stakeholders throughout the research project. This will in turn help ensure that the project is centred on a common goal, with clear understandings of the different research activities and potential impact. This can also promote research uptake and enable researchers to make adjustments to project activities, maximizing stakeholder engagement and research utilization.

M&E aims & objectives

The overarching aim of the monitoring and evaluation work package is to monitor and evaluate the various GELA project activities and processes, including whether, how and why activities took place or if goals were met.

The specific monitoring and evaluation objectives are to:

Monitor ongoing engagement with local stakeholders across work packages and explore what worked and didn’t and why;

Assess the capacity development needs of guideline panels and steering group committees and explore their views and experiences of the project’s capacity development activities;

Explore guideline panelists’ experiences with reading and using evidence from reviews of qualitative research, including their preferences regarding how qualitative review findings are summarized and presented;

Evaluate guideline panelists’, steering group committees’ and project team members’ overall views and experiences of the project, including the what works or not, to influence evidence-informed decision-making and guideline adaptation processes

Overall approach

We will use a longitudinal, mixed-methods study design, informed by an overarching project Theory of Change (Table  2 ). The theoretical underpinning for the GELA project across all work packages is related to the three-layered behaviour change wheel comprising opportunity, capability and motivation [ 42 ]. The design, delivery and implementation of multi-stakeholder integrated activities based on identified priority areas and needs is expected to lead to guideline related improved capacity, practice and policy within each country’s health system. Certain objectives also have specific underpinning theoretical frameworks, in addition to the overarching project Theory of Change, which are explained under the respective objectives below. A series of interconnected qualitative and quantitative data collections methods will be used to address each objective.

In what follows, we describe each objective and the methods we will use to achieve it, separately. However, in many cases the qualitative data collection cuts across objectives, with the same interviews and observations being used to explore multiple issues simultaneously (e.g. knowledge translation, capacity, overall views and experiences of the project, etc.). The relationship between the different objectives and associated methods are depicted in Tables 3 and 4 . Table 3 outlines the stakeholder groups included in the monitoring and evaluation work package, including their composition and for which objectives they are targeted. Table 4 provides the timeline for the different data collection methods and how they relate to each across the objectives.

1. Objective 1: monitor ongoing engagement with local stakeholders across work packages and explore what worked and did not work and why

Overall approach for this objective.

This objective will be guided by an integrated knowledge translation (IKT) approach. IKT focuses on the important role of stakeholder engagement in enhancing evidence-informed decision-making [ 43 ]. As part of work package 4 (‘dissemination and communication’), knowledge translation (KT) champions have been identified in each of the three countries and will work together to develop and implement country-level KT strategies. This will include defining KT objectives, identifying and mapping relevant stakeholders, prioritizing those we will actively engage and developing a strategy for engaging each priority stakeholder. We will monitor these engagements through the development and implementation of a tracking sheet, qualitative case studies and semi-structured interviews.

Participants

Participants will comprise of knowledge translation (KT) champions and relevant country-level stakeholders. KT champions are GELA project staff who have dedicated time to work on the communication, dissemination and engagement aspects at a country-level. At least one KT champion has been identified for each of Malawi, Nigeria and South Africa.

Relevant country-level stakeholders will be identified as part of the KT strategy development (WP4) and will comprise any health decision-makers, e.g. health practitioners, community groups, health system managers, policy-makers, researchers and media.

Tracking sheet and qualitative case studies

A tracking sheet will be used to capture information for each stakeholder related to the purpose, message, medium or forum, messenger, timing and resources for engagement. KT champions in each country will be responsible for tracking these details on a continuous basis, and the tracking sheet will be monitored bi-monthly at a meeting with KT champions from the three country teams. This will help us monitor whether and how engagement activities are taking place, as well as the strategies for implementation. The tracking sheets will consist of different in-country stakeholders (e.g. government officers, health professional associations, researchers, media, etc.), and there may be several goals for engaging each individual stakeholder. The engagement strategy will be reviewed and updated as priority stakeholders change over the research stages and project period. As such, the sample size will be determined iteratively.

We will analyse information with descriptive statistics. For example, we will group and count by categories: number and type of stakeholders, type of engagement activities, type of KT products produced, type of forum or medium used for dissemination, frequency and duration of engagement, follow-ups, intensive engagement period and resources required for engagement.

We will also develop case stories (or impact stories) describing engagement activities and processes between project staff and relevant stakeholders. The case studies will help us monitor successful engagement, disseminate best practice scenarios and draw out lessons for future engagements. We will identify case stories through the tracking sheet and at bi-monthly meetings with the KT co-ordinator, where KT champions will be asked to share success stories or learning moments. KT champions will not know which ‘case’ will be selected for the case study in advance. The information will be collected by the KT co-ordinator, who is not involved in any of the country strategy implementation. The information collected from the KT champions (and messenger, if the messenger is not the KT champion) will be via a standard case story template, including aim of engagement, what the engagement was, experiences from both sides (quotes to be included in stories), success of engagement, lessons learnt and any future engagement plans. The number of cases will be determined iteratively. The intention is to develop one case story from each country annually, showcasing different cases, e.g. type of KT goal, type of stakeholder, type of KT medium/forum, etc.

Semi-structured interviews

At project close (month 30), we will conduct semi-structured interviews to explore if, why and how project KT goals were met and what planned stakeholder engagements worked (and did not work) and why. The interviews will be conducted with KT champions, other messengers (e.g. communication officers), country leads and selected stakeholders. At least two people from each county (KT champion and messenger and/or stakeholder) will be interviewed, and so there will be six to eight interviews in total. Participants will be selected purposively for information-rich cases that can help yield insights and in-depth understanding of the nature and success (or not) of our stakeholder engagements [ 44 ].

These interviews will form part of the interviews conducted with project team members more broadly as part of objective 4, the methods of which are therefore described in more detail below.

2. Objective 2: assess the capacity development needs of guideline panels and steering group committees and explore their views and experiences of the project’s capacity development activities.

Overarching theoretical lens.

We will draw on the Kirkpatrick model [ 45 ] as the underpinning theoretical framework for this objective. This model evaluates training effectiveness across four levels: (1) reaction, (2) learning, (3) behaviour and (4) results. The ‘reaction level’ assesses the degree of satisfaction of participants with the training event. The ‘learning level’ examines learning among participants both before and after the training event to determine any change in knowledge [ 46 , 47 ]. The ‘behaviour level’ assesses whether the training event has provided any favourable change in behaviour among participants. The final ‘results level’ assesses the use of knowledge gained through the training event within the workplace [ 46 , 47 ].

To assess the potential difference that project capacity development activities make, the outcomes of interest will be those related to training in evidence-based healthcare (EBHC). An overview of systematic reviews by Young and colleagues identified that EBHC training often aims to ‘improve critical appraisal skills and integration of results into decisions, and improved knowledge, skills, attitudes and behaviour among practising health professionals’ [ 48 , 49 ].

We will employ mixed methods to achieve this objective, including three rounds of online surveys (at baseline, mid-line and at the project close) as well as semi-structured interviews (at project close) and non-participant observations of meetings (various). The first online survey at baseline will assess the capacity needs of the guideline panels and steering group committees in South Africa, Malawi and Nigeria, and the two subsequent online surveys will assess the potential difference project capacity development activities make on these groups across all the four levels of the Kirkpatrick model, i.e. reaction, learning, behaviour and results. The capacity needs and progress of these groups will also be explored qualitatively through semi-structured interviews and observations of meetings.

Details of the project capacity development activities that will be implemented as part of work package 5 (‘capacity strengthening and sharing’) of the GELA project are outlined in Table  1 (above). All members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria will be invited and encouraged to attend all project capacity development activities. ‘On the job’ capacity building will also take place during the various meetings convened with these groups, as they are supported to identify priority topics, to appraise and discuss the evidence used to inform the recommendations and to formulate the final recommendations.

Participants will comprise members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria. Table 3 (above) provides details of the composition of the guideline panels and steering group committees.

Online surveys

Procedures and data collection tools.

At baseline (at approximately 6 months before engagement in any project training activities), at mid-line (month 18) and at the project close (month 30), all members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria will be invited, via email, to participate in a survey. In each of the three countries the guideline development group and steering group committees will include approximately 20 and 10 members, respectively; we will therefore aim to have 90 participants in total complete the survey. The email invitation to all three survey rounds will inform participants about the nature of the study and direct them to an online survey. The landing page of the survey will provide information about the purpose of the research project and what is being requested from the participants, with a consent statement at the end which the participant will be required to agree to before being able to continue with the survey. Data will only be collected from participants who consent to freely participate in the study. The survey will be carried out using a secure online survey platform (such as Microsoft Forms) where all cookies and IP address collectors will be disabled to protect the confidentiality of the participants and to avoid tracking of the participant activities online. Unique identifiers (last six numbers of their ID) will be used to track participants responses over time and link data from baseline to project close.

The baseline survey will be a short (10–25 min) form that will ask participants about their capacity needs and knowledge/skills in evidence-based healthcare (EBHC) and decision-making. The survey will capture demographic variables of participants at baseline, mid-term and at the end of the project. It will assess the training needs of participants at baseline, participants’ satisfaction at the end of each training activity, the knowledge and skills at baseline, mid-term and at the end of the project. Participants’ behaviour will also be assessed using open-ended questions and vignettes. The surveys will focus on all four levels (i.e. reaction, learning, behaviour and results) of the Kirkpatrick model.

Data management and analysis

All data collected on the secure online survey platform will be coded, cleaned and entered into STATA. Data collected for the baseline survey will be analysed using descriptive statistics to determine the frequency of the various training needs and qualitative data gathered using the open-ended questions will be analysed thematically using manual coding (or if available and dataset is large), and NVivo or a similar tool will be used to identify the recurring themes which emerge in the data collected about the key training needs of participants.

Data collected for the surveys conducted at midpoint and at project close will be analysed using descriptive statistics to determine if there has been a change in the learning, knowledge gained and behaviours over time, as well as the extent of the potential application of evidence-based practice, while the data collected using the open-ended questions will be analysed using thematic analysis outlining how project capacity development activities informed particular outcomes and results in the participant’s workplace. To determine change in skills (and trends over time such as confidence improvement or decay), the descriptive statistics will be supplemented by appropriate inferential statistics for repeated measures (paired data) such as McNemar or paired t -tests, reporting change in percentages as mean differences (such as self-reported confidence) with 95% confidence intervals or/and frequencies. Descriptive trends over time will also be presented graphically using line graphs or other visual aids as appropriate. However, these will be interpreted with caution as the primary analysis is descriptive. Statistical significance will be set at a p value of 0.05.

At project close (month 30), we will conduct semi-structured interviews with a sample of members from the guideline panels and steering group committees in South Africa, Malawi and Nigeria. Sampling will be purposive, with the aim of understanding the broad range of needs, experiences and perspectives and ensuring that the sample reflects a range of socio-demographic characteristics and stakeholder categories. We will begin with a sample size of 10–15 participants in each country; however, sampling will continue if we have not reached saturation of the data through the initial sample size [ 44 ].

Participants will be contacted, either by telephone or via email, and invited to participate in an interview. Interviews will be conducted face-to-face or electronically (e.g. using Microsoft Teams) at a date and time chosen by participants. Face-to-face interviews will take place at a location convenient to participants, which is conducive to a confidential exchange. The interviews will last between 45 and 60 min and will be conducted by researchers trained in qualitative research methodologies and interviewing techniques. The interviews will be guided by a semi-structured topic guide and will include questions informed by the four levels (i.e. reaction, learning, behaviour and results) of the Kirkpatrick model. Specifically, the questions will explore participants’ views and experiences regarding their capacity development needs and expectations of the project; whether and why these expectations were met (or not), the project capacity development activities, what they learned (or not) from these activities and what impact participants believe they have had (or may have) on their practices.

Verbal and written information about the study will be provided to all participants taking part in interviews. Written informed consent will be obtained from all participants before proceeding with the interview. With the permission of participants, all interviews will be digitally recorded.

Non-participant observations

We will conduct non-participant observations of guideline panel and steering group committee meetings. Observational methods can provide useful data on what people do, how they interact with each other and how they engage with particular artefacts in situ (rather than their accounts of these) [ 50 ]. The steering group committees in each country will meet approximately twice over the project duration (with the option for additional meetings): an initial meeting for project orientation (month 2/3) and again to identify priority topics and guideline gaps (month 6). Guideline panels in each country will meet approximately three times over the project duration (with the option for additional meetings): an initial meeting for project orientation and outcome prioritization (month 6/7), another potential meeting if necessary to finalize outcome prioritization and a final meeting to draft recommendations for the guideline (months 17–20). Meetings for both groups will be held virtually or in person, informed by preferences of the committee.

With the exception of the initial steering group committee (month 2/3), at least one researcher will be present to observe guideline panel and steering group committee meetings. The observer will aim to identify any capacity-related needs, expectations, gaps, strengths, achievements and challenges and the contexts in which these occur. He or she will also pay particular attention to group dynamics and the interactions between members and different stakeholder groups, and the potential impact of these on capacity-related issues. Observations will be informed by Lofland’s [ 51 ] criteria for organizing analytical observations (acts, activities, meanings, participation, relationships and settings). The observer will take detailed observational notes. With consent of the attendees, all meetings will also be digitally recorded. The recordings will be used to identify further issues not identified and to deepen or clarify issues noted, through the real-time observations of verbal engagements.

Data management and analysis: semi-structured interviews and observations

Interview and meeting recordings will be transcribed verbatim, and all personal identifying information will be removed from transcripts. The anonymized transcripts, together with observational notes, will be downloaded into Nvivo, a software programme that aids with the management and analysis of qualitative data. Analysis of the qualitative data will proceed in several rounds. First, as with all qualitative data analysis, an ongoing process of iterative analysis of the data will be conducted throughout the data collection period. Second, we will use a thematic analysis approach, using the phases described by Braun and Clarke [ 52 ], to identify key themes pertaining to participants’ capacity development needs and expectations and whether, how and why project capacity development activities met (or not) these needs and expectations. Finally, findings from the surveys (as described above) will also be integrated with the findings from the thematic analysis using a ‘narrative synthesis’ approach, a technique recommended by the Cochrane Collaboration as a way of synthesizing diverse forms of qualitative and quantitative evidence in mixed methods studies [ 53 , 54 ]. This approach will allow for both robust triangulation, and a more comprehensive interpretation of the difference project capacity development activities may have made on the guideline panels and steering group committees.

3. Objective 3: explore guideline panelists’ experiences with reading and using evidence from reviews of qualitative research, including their preferences regarding how qualitative review findings are summarized and presented.

Objective 3 of the monitoring and evaluation stakeholder matrix work package explores how guideline panels view and experience evidence from the review(s) of qualitative research, including how it is summarized and presented. Here, we will employ a user testing approach, drawing on the methods and guidance of the SURE user test package 2022 developed by Cochrane Norway ( https://www.cochrane.no/our-user-test-package ) and which has been used to test various evidence-related products [ 55 , 56 , 57 , 58 ]. User testing involves observing people as they engage with a particular product and listening to them ‘think-aloud’. The goal is to gain an understanding of users’ views and experiences, the problems they face and to obtain suggestions for how a product may be improved [ 55 , 56 , 57 , 58 ].

We will begin by identifying or preparing relevant reviews of qualitative research. We will then develop review summary formats and explore guideline panel members’ views and experiences of these formats. We will revise the formats in multiple iterative cycles.

Identifying or preparing relevant reviews of qualitative research

As part of WP2 of the project (‘evidence synthesis’), we will identify relevant review(s) of qualitative research, including reviews exploring how people affected by the interventions of interest value different outcomes, the acceptability and feasibility of the intervention and potential equity, gender and human rights implications of the intervention. These reviews need to be assessed as sufficiently recent and of a sufficient quality. They also need to have applied GRADE-CERQual assessments to the review findings. Where necessary, we will update existing reviews or prepare reviews ourselves.

Developing the review summaries

In WP3 of the project (‘decision-making’) the evidence from these reviews will be provided to guideline panels as part of the evidence-to-decision (‘EtD’) frameworks that will inform the recommendations they develop (see Table  1 for further details about project work packages 2 and 3). Our next step will therefore be to prepare summaries of the reviews in a format that can easily be included in the EtD frameworks.

Each summary needs to present review findings that are relevant to specific parts of the EtD framework (typically the ‘values’, ‘acceptability’, ‘feasibility’ and ‘equity’ components). It also needs to include information about our confidence in these findings. Finally, the summary needs to indicate where this evidence comes from and to allow guideline panels to move from the summary to more detailed information about the evidence.

Most of this information is found in the review’s Summary of Qualitative Findings tables. However, these tables are usually too large for EtD frameworks and are not tailored to each framework component. We will, therefore, start by creating new summaries, using a format that we have previously used in EtD frameworks [ 59 , 60 , 61 ] but that we have not user tested. As opposed to the Summary of Qualitative Findings tables, where each finding and our confidence in the finding, is presented individually in separate rows, this format involves pulling the findings and confidence assessments together in short, narrative paragraphs.

User testing the summary format

For our first set of user tests, we will observe guideline panels participating in the CPG panel simulation workshops. For our second round of user tests, we will observe how the guideline panels experience and interact with this qualitative evidence during the real guideline processes. Third, we will then test a potentially refined format with a selection of guideline panel members using a semi-structured interview guide. Finally, at the end of the project, we will conduct semi-structured interviews with a selection of guideline panel members to explore their broader views and experiences of interpreting and using evidence from reviews of qualitative studies in their deliberation processes. Figure  1 provides a visual depiction of this iterative process.

Iterative approach for user testing evidence from reviews of qualitative research

We will draw on the adapted version of Peter Morville’s original honeycomb model of user experience [ 62 ] as the underpinning theoretical framework for this objective [ 63 ] (Fig.  1 ). This adapted version extends and revises the meaning of the facets of user experience depicted in the original model. It includes eight facets: accessibility, findability, usefulness, usability, understandability, credibility, desirability and affiliation. Accessibility involves whether there are physical barriers to gaining access; findability is about whether the person can locate the product or the content that they are looking for; usefulness is about whether the product has practical value for the person; usability comprises how easy and satisfying the product is to use; understandability is about whether the person comprehends correctly both what kind of product it is and the content of the product (and includes both user's subjective perception of her own understanding and an objective measure of actual/correct understanding); credibility comprises whether the product/content is experienced as trustworthy; desirability is about whether the product is something the person wants and has a positive emotional response to it; affiliation involves whether the person identifies with the product, on a personal or a social level, or whether it is alienating and experienced as being not designed for ‘someone like me’. The adapted model also adds to the original model a dimension of user experience over time, capturing the chronological and contingent nature of the different facets.

Participants will comprise members of the guideline panels in South Africa, Malawi and Nigeria. Table 3 (above) provides details of the composition of the guideline panels.

Non-participant observations: guideline panel simulation workshops and guideline panel meetings

We will conduct non-participant observations of the CPG panel simulation workshops and the subsequent guideline panel meetings for developing the recommendations. The CPG panel simulation workshops will run a simulation of a real guideline process and give guideline panels an opportunity to understand how the guideline process works before they participate in real panel meetings. The guideline panels in all three countries will be invited and encouraged to attend these workshops, which will form part of the project capacity development activities of WP5 (Table  1 ).

With the participants’ consent, both the simulation workshops and meetings will be digitally recorded and at least two observers will observe and take notes. The observations will focus on how guideline panel members refer to and interact with the summaries of qualitative evidence. Drawing on a user testing approach ( https://www.cochrane.no/our-user-test-package ), we will also look specifically for both problems and facilitators in the way the qualitative evidence is formatted, including ‘show-stoppers’ (the problem is so serious that it hindered participants from correct understanding or from moving forward), ‘big problems/frustrations’ (participants were confused or found something difficult but managed to figure it out or find a way around the problem eventually), ‘minor issues/cosmetic things’ (small irritations, frustrations and small problems that do not have serious consequences, as well as likes/dislikes), ‘positive/negative feedback’, ‘specific suggestions’, ‘preferences’ and any other ‘notable observations’, e.g. feelings of ‘uncertainty’.

Structured user testing interviews

Based on the insights gained from the non-participant observations (above), we may make changes or refinements to our original summary format (Fig.  1 ). Once the guideline panel meetings have concluded (approximately by month 20), we will then conduct structured user testing interviews to test the potentially refined summary format. These interviews will be conducted with a sample of members from the guideline panels in South Africa, Malawi and Nigeria. Sampling will be purposive, with the aim of understanding the broad range of experiences and perspectives and ensuring the sample reflects a range of socio-demographic characteristics and stakeholder categories. As recommended ( https://www.cochrane.no/our-user-test-package ), we will begin with a sample size of six to eight participants in each country; however, sampling will continue until saturation is achieved [ 44 ].

Participants will be contacted, either telephonically or via email, and invited to participate in an interview. Interviews will be conducted face-to-face or electronically (e.g. using Skype or Teams) at a date and time chosen by participants. Face-to-face interviews will take place at a location convenient to participants, which is conducive to a confidential exchange. In line with the SURE user test package 2022 guidance, the interviews will last approximately 60 min ( https://www.cochrane.no/our-user-test-package ). They will be facilitated by a test leader, who will accompanied by at least one observer who will take notes. Both the test leader and observer(s) will be trained in user testing interviewing methodology and techniques. Verbal and written information about the study will be provided to all participants taking part in interviews. Written informed consent will be obtained from all participants before proceeding with the interview. With the permission of participants, all interviews will be video recorded.

For these interviews we will show panel members the latest version of the format, explore immediate first impressions, and then opinions about different elements of the summary. We may also show panel members different formats where we think this may be helpful. We will use a structured interview guide which draws heavily on other interview guides that been developed to user test evidence-related products [ 55 , 56 , 57 , 58 ]. It will include questions related the participant’s background; their immediate first impressions of the summary format(s); in-depth walk-through of the summary format(s), with prompts to think aloud what they are looking at, thinking, doing and feeling; and suggestions for improving the way the summary is formatted and for improving the user testing itself. We may ask follow-up questions to specific issues we observed in the simulation workshops and guideline panel meetings and/or create scenarios that resemble issues we observed in the workshops/meetings. This will be decided upon based on the findings that emerge from these workshops/meetings. The guide will be finalized once the relevant qualitative evidence (from WP2) has been produced and we have gained insights from the workshops and meetings.

As with the non-participant observations of meetings and workshops, throughout the interview, the observers will make notes about the participant’s experience as heard, observed and understood. Drawing on a user testing approach, they will look specifically for both problems and facilitators, specific suggestions, preferences and any other notable observations (as described above under ‘non-participant observations’).

At project close (month 30), we will also conduct semi-structured interviews with a sample of members from the guideline panels in South Africa, Malawi and Nigeria. These will be the same interviews with guideline panel members as described in objective 2. In addition to exploring participants’ capacity development needs, expectations and achievements, the semi-structured topic guide will also explore their views and experiences of (and specific capacity in) interpreting and using evidence from reviews of qualitative studies in guideline processes. More specifically, questions will investigate participants’ familiarity/experience with qualitative evidence; their perceptions of different types of evidence, what constitutes qualitative evidence and the role of qualitative evidence in guideline processes; and their experiences of using the qualitative evidence in their deliberations as part of the project, including what influenced its use and whether they found it useful. Details pertaining to sampling, data collection procedures and collection tools are described in objective 2.

All interview and meeting recordings will be transcribed verbatim, and all personal identifying information will be removed from transcripts. The anonymized transcripts, together with observational notes (from the workshops, meetings and interviews), will be downloaded into a software programme that aids with the management and analysis of qualitative data. Analysis of the data will be guided by the user testing analysis methods described in the SURE user test package 2022 ( https://www.cochrane.no/our-user-test-package ). The analysis will proceed in several, iterative rounds to develop and revise the summary format and to inform the focus of subsequent data collection. After each user test, we will review our notes, first separately and then together. In line with the SURE user test package 2022 guidance, we will look primarily for barriers and facilitators related to correct interpretation of the summary’s contents, ease of use and favourable reception, drawing on the facets of the revised honeycomb model of user experience (Fig.  2 ). We will trace findings back to specific elements or characteristics of the summaries that appeared to facilitate or hinder problems. Before the next set of user tests, we will discuss possible changes that could address any identified barriers and make changes to the summary format.

Adapted version of Peter Morville’s honeycomb model of user experience

4. Objective 4: evaluate guideline panelists’, steering group committees’ and project team members’ overall views and experiences of the project, including what works or not, to influence evidence-informed decision-making and guideline adaptation processes.

This objective explores overall views and experiences of the project, with a focus on guideline panelists, steering group committees and project team members. Specifically, it seeks to gain an understanding of these three stakeholder groups’ more general views and experiences of the project activities they were involved with and whether, why and how these activities may influence (or not) evidence-informed decision-making and guideline adaptation processes. This will be achieved through semi-structured interviews.

Participants will comprise members of the guideline panels and steering group committees in South Africa, Malawi and Nigeria, as well as members of the project team (as described in Table  3 above).

At project close (month 30), we will conduct semi-structured interviews with a sample of members from the guideline panels and steering group committees in South Africa, Malawi and Nigeria. These will be the same interviews and participants as described in objective 2. In addition to exploring issues around capacity development and qualitative evidence, the interviews will also investigate participants’ views and experiences of the various project activities they were involved with, and whether, why and how these activities may influence (or not) evidence-informed decision-making and guideline adaptation processes. Details pertaining to sampling, data collection procedures and collection tools are described in objective 2.

At project close (month 30), we will also conduct semi-structured interviews with members of the project team (see Table  3 for details of project team composition). We will begin by interviewing all project management team members, WP leads and KT champions. Additional participants will be determined iteratively (depending on what emerges from initial interviews) and purposively, with the aim of understanding the broad range of experiences and perspectives and ensuring the sample reflects the various groups which make up the project team. Interviews will be conducted face-to-face or electronically (e.g. using Skype or Teams) at a date and time chosen by the interviewee. The interviews will last between 45 and 60 min and will be guided by a semi-structured topic guide. The questions will explore participants’ views and experiences of the respective work packages in which they were involved, including what the primary goals of the work package were; if, why and how these goals were met; and what worked and what did not work and why.

The same qualitative data analysis procedures and methods will be used as described in objective 2. For this objective, the thematic analysis will identify key themes pertaining to views and experiences of project activities, including what worked (or not) and why, whether, why and how the project may (or not) influence evidence-informed decision-making and guideline development, adaptation and dissemination processes in South Africa, Malawi and Nigeria and potential barriers and facilitators to the sustainability of this influence.

Evidence-based guideline development is a multi-stakeholder, multi-perspective, complex set of tasks. There is limited, if any, research that has followed these steps from the perspectives of policymakers or researchers from start to end. The GELA project protocol sets out to monitor and evaluate various key steps in the process, using in-depth qualitative methods alongside appropriate surveys not only to inform the project as it progresses but also to understand the overall impact of all steps on development of transparent and contextually-rich guideline recommendations. Following WHO’s guideline steps, the tasks range from scoping stakeholder-informed priority topics to conducting relevant data gathering and evidence synthesis, followed by guideline panel meetings to reach consensus decisions and finally to produce recommendations that can be useful to end-users and improve health and care outcomes. The GELA project is undertaking a 3-year project to conduct these tasks in the context of newborn and child health priorities. We are doing this in collaboration with national ministries of health, academics, non-governmental partners and civil society groups in Malawi, Nigeria and South Africa. Overall, we aim build capacity across all collaborators for evidence-informed guideline development, while producing fit for context guideline recommendations, in accessible formats that benefit children, caregivers and health care providers.

As such, this is a practical research project, in that the products should directly impact care decisions at the national level but with the added benefit of being able to learn about what works or does not work for collaborative guideline development in country. We will also be applying emergent guideline adaptation methods to explore reducing duplication of expensive guideline development efforts in our lower resource settings. Our project addresses newborn and child health, keeping this most vulnerable population in our focus, hoping that producing sound evidence-based recommendations has the potential to impact care.

Through some of our formative work, we have completed a landscape analysis identifying and describing all available newborn and child health guidelines in each of the partner countries. In all countries there were similar findings, (1) there is no easy access to guidelines for end-users, thus locating a guideline requires effort and screening through multiple sources; (2) considering national priority conditions in this age group, there were often gaps in available current guidelines for managing children; and (3) when we appraised the guidelines using the global standard, AGREE II tool, we found that the reporting of guideline methods were poor, leaving it uncertain whether the recommendations were credible or whether any influences or interests had determined the direction of a recommendation. Finally, we expected to find many adapted guidelines, based on WHO or UNICEF or similar guidance available globally; however, very few of the identified guidelines stated clearly whether they had been adapted from other sources and, if so, which recommendations were adopted and which adapted.

Given progress globally in methods for guideline development, the continued poor reporting on guideline methods at the country level speak to a breakdown in skills-sharing globally, for example, WHO produces guidelines that are recognized as rigorous and follow good practice and reporting, but the same standards are not supported in country. Overall, GELA aims to address these key gaps in national guideline approaches for adaptation, but we need to recognize that this will be a long term process and that we need to learn from each other about what works and what may not serve us. Therefore, this protocol outlines our approach for monitoring several aspects of the project in our efforts to move closer to trustworthy and credible guidelines that all can use and trust for countries like ours.

Availability of data and materials

Not applicable.

Abbreviations

Poverty-related diseases

Sub-Saharan Africa

• Clinical practice guidelines

Evidence-informed decision-making

Evidence-based healthcare

Global Evidence Local Adaptation

Knowledge translation

Levels and trends in child mortality: report 2020. Estimates developed by UNICEF. ISBN: 978-92-806-5147-8. https://www.who.int/publications/m/item/levels-and-trends-in-child-mortality-report-2020 . United Nations Children’s Fund. Accessed Aug 2024.

Liu L, Oza S, Hogan D, et al. Global, regional, and national causes of under-5 mortality in 2000–15: an updated systematic analysis with implications for the sustainable development goals. Lancet. 2016;388(10063):3027–35.

Article   PubMed   PubMed Central   Google Scholar  

Under-five mortality. 2021 https://data.unicef.org/topic/child-survival/under-five-mortality/ . United Nations International Children’s Fund. Accessed Aug 2024.

Stewart R, El-Harakeh A, Cherian SA. Evidence synthesis communities in low-income and middle-income countries and the COVID-19 response. Lancet. 2020;396(10262):1539–41.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mijumbi RM, Oxman AD, Panisset U, Sewankambo NK. Feasibility of a rapid response mechanism to meet policymakers’ urgent needs for research evidence about health systems in a low income country: a case study. Implementation Sci. 2014;9:114.

Article   Google Scholar  

Stewart R, Dayal H, Langer L, van Rooyen C. The evidence ecosystem in South Africa: growing resilience and institutionalisation of evidence use. Palgrave Commun. 2019;5(1):90.

Uneke CJ, Ezeoha AE, Ndukwe CD, Oyibo PG, Onwe F. Promotion of evidence-informed health policymaking in Nigeria: bridging the gap between researchers and policymakers. Glob Public Health. 2012;7(7):750–65.

Article   PubMed   Google Scholar  

Young T, Garner P, Clarke M, Volmink J. Series: clinical epidemiology in South Africa Paper 1: evidence-based health care and policy in Africa.: past, present, and future. J Clin Epidemiol. 2017;83:24–30.

Vandvik PO, Brandt L. Future of evidence ecosystem series: evidence ecosystems and learning health systems: why bother? J Clin Epidemiol. 2020;123:166–70.

Graham R, Mancher M, Wolman D, Greenfield S, Steinberg E. Clinical practice guidelines we can trust. Washington, DC: The National Academy Press; 2011.

Book   Google Scholar  

Kredo T, Bernhardsson S, Machingaidze S, et al. Guide to clinical practice guidelines: the current state of play. Int J Qual Health Care. 2016;28(1):122–8.

McCaul M, Tovey D, Young T, et al. Resources supporting trustworthy, rapid and equitable evidence synthesis and guideline development: results from the COVID-19 evidence network to support decision-making (COVID-END). J Clin Epidemiol. 2022;151:88–95.

Tricco AC, Garritty CM, Boulos L, et al. Rapid review methods more challenging during COVID-19: commentary with a focus on 8 knowledge synthesis steps. J Clin Epidemiol. 2020;126:177–83.

World Health Organization. WHO handbook for guideline development. Geneva: World Health Organization; 2011.

Google Scholar  

Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186(3):E123–42.

Kredo T, Gerritsen A, van Heerden J, Conway S, Siegfried N. Clinical practice guidelines within the Southern African development community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases. Health Res Policy Syst. 2012;10:1.

Malherbe P, Smit P, Sharma K, McCaul M. Guidance we can trust? The status and quality of prehospital clinical guidance in sub-Saharan Africa: a scoping review. African J Emerge Med. 2021;11(1):79–86.

McCaul M, Clarke M, Bruijns SR, et al. Global emergency care clinical practice guidelines: a landscape analysis. African J Emerge Med. 2018;8(4):158–63.

Dizon JM, Grimmer K, Louw Q, Kredo T, Young T, Machingaidze S. South African guidelines excellence (SAGE): adopt, adapt, or contextualise? South African Med J. 2016;106(12):1177–8.

Article   CAS   Google Scholar  

Schünemann HJ, Wiercioch W, Brozek J, et al. GRADE evidence to Decision (EtD) frameworks for adoption, adaptation, and de novo development of trustworthy recommendations: GRADE-ADOLOPMENT. J Clin Epidemiol. 2017;81:101–10.

ADAPTE Collaboration. 2009 The ADAPTE process: Resource toolkit for guideline adaptation, version 2. http://www.g-i-n.net/ . Accessed Aug 2024.

Barreix M, Lawrie TA, Kidula N, et al. Development of the WHO antenatal care recommendations adaptation toolkit: a standardised approach for countries. Health Res Policy Syst. 2020;18(1):70.

Ioannidis JP. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.

Opiyo N, Shepperd S, Musila N, English M, Fretheim A. The, “child health evidence week” and grade grid may aid transparency in the deliberative process of guideline development. J Clin Epidemiol. 2012;65(9):962–9.

Wang Z, Norris SL, Bero L. The advantages and limitations of guideline adaptation frameworks. Implemen Sci. 2018;13(1):72.

Langlois EV, Tunçalp Ö, Norris SL, Askew I, Ghaffar A. Qualitative evidence to improve guidelines and health decision-making. Bull World Health Organ. 2018. https://doi.org/10.2471/BLT.17.206540 .

Carmona C, Baxter S, Carroll C. Systematic review of the methodological literature for integrating qualitative evidence syntheses into health guideline development. Res Synth Methods. 2021;12(4):491–505.

Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: Literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Res Alloc. 2012;10(1):9.

Verkerk K, Van Veenendaal H, Severens JL, Hendriks EJ, Burgers JS. Considered judgement in evidence-based guideline development. Int J Qual Health Care. 2006;18(5):365–9.

Alonso-Coello P, Schunemann HJ, Moberg J, et al. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ. 2016. https://doi.org/10.1136/bmj.i2016 .

Glenton C, Lewin S, Gülmezoglu AM. Expanding the evidence base for global recommendations on health systems: strengths and challenges of the OptimizeMNH guidance process. Implement Sci. 2016;11:98.

Lewin S, Glenton C. Are we entering a new era for qualitative research? using qualitative evidence to support guidance and guideline development by the World Health Organization. Int J Equity Health. 2018;17(1):126.

Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Chapter 21: Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook . Accessed Aug 2024.

Glenton C, Lewin S, Norris S. Using evidence from qualitative research to develop WHO guidelines (Chapter 15) World Health Organization Handbook for Guideline Development. 2nd ed. WHO: Geneva; 2016.

World Health Organisation. WHO recommendations: intrapartum care for a positive childbirth experience. Geneva: WHO; 2018.

Rosenbaum SE, Glenton C, Oxman AD. Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information. J Clin Epidemiol. 2010;63(6):620–6.

Lewin S, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a summary of qualitative findings table. Implement Sci. 2018;13(Suppl 1):10.

Treweek S, Oxman AD, Alderson P, et al. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci. 2013;8:6.

Brandt L, Vandvik PO, Alonso-Coello P, et al. Multilayered and digitally structured presentation formats of trustworthy recommendations: a combined survey and randomised trial. BMJ Open. 2017;7(2): e011569.

Kredo T, Cooper S, Abrams AL, et al. ’Building on shaky ground’-challenges to and solutions for primary care guideline implementation in four provinces in South Africa: a qualitative study. BMJ Open. 2020;10(5): e031468.

Vandvik PO, Brandt L, Alonso-Coello P, et al. Creating clinical practice guidelines we can trust, use, and share: a new era is imminent. Chest. 2013;144(2):381–9.

Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011;6:42.

Jessani NS, Rohwer A, Schmidt B-M, Delobelle P. Integrated knowledge translation to advance noncommunicable disease policy and practice in South Africa: application of the exploration, preparation, implementation, and sustainment (EPIS) framework. Health Res Policy Syst. 2021;19(1):82.

Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229–45.

Roos M, Kadmon M, Kirschfink M, et al. Developing medical educators–a mixed method evaluation of a teaching education program. Med Educ Online. 2014;19:23868.

Bijani M, Rostami K, Momennasab M, Yektatalab S. Evaluating the effectiveness of a continuing education program for prevention of occupational exposure to needle stick injuries in nursing staff based on Kirkpatrick’s model. J Natl Med Assoc. 2018;110(5):459–63.

PubMed   Google Scholar  

Carlfjord S, Roback K, Nilsen P. Five years’ experience of an annual course on implementation science: an evaluation among course participants. Implement Sci. 2017;12(1):101.

Young T, Rohwer A, Volmink J, Clarke M. What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLoS ONE. 2014;9(1): e86706.

Young T, Dizon J, Kredo T, et al. Enhancing capacity for clinical practice guidelines in South Africa. Pan Afr Med J. 2020;36:18.

Green G, Thorogood N. Qualitative methods for health research. London: Sage; 2004.

Lofland J. Analyzing social settings. Belmont, CA: Wadsworth Publishing Company; 1971.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101.

Popay J, Roberts H, Sowden A. Guidance on the conduct of narrative synthesis in systematic reviews a product from the ESRC methods programme. I Health Research: Lancaster; 2006.

Pope C, Mays N, Popay J. How can we synthesize qualitative and quantitative evidence for healthcare policy-makers and managers? Healthc Manage Forum. 2006;19(1):27–31.

Glenton C, Santesso N, Rosenbaum S, et al. Presenting the results of Cochrane systematic reviews to a consumer audience: a qualitative study. Med Dec Making. 2010;30(5):566–77.

Rosenbaum SE, Glenton C, Cracknell J. User experiences of evidence-based online resources for health professionals: user testing of The Cochrane library. BMC Med Inform Decis Mak. 2008;8:34.

Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. J Clin Epidemiol. 2010;63(6):607–19.

Rosenbaum SE, Glenton C, Wiysonge CS, et al. Evidence summaries tailored to health policy-makers in low- and middle-income countries. Bull World Health Organ. 2011;89(1):54–61.

Downe S, Finlayson KW, Lawrie TA, et al. Qualitative evidence synthesis (QES) for guidelines: Paper 1 - using qualitative evidence synthesis to inform guideline scope and develop qualitative findings statements. Health Res Policy Syst. 2019;17(1):76.

Lewin S, Glenton C, Lawrie TA, et al. Qualitative evidence synthesis (QES) for guidelines: Paper 2 - using qualitative evidence synthesis findings to inform evidence-to-decision frameworks and recommendations. Health Res Policy Syst. 2019;17(1):75.

Glenton C, Lewin S, Lawrie TA et al. Qualitative Evidence Synthesis (QES) for Guidelines: Paper 3 – Using qualitative evidence syntheses to develop implementation considerations and inform implementation processes. Health Res Policy Sys. 2019;17:74. https://doi.org/10.1186/s12961-019-0450-1

Morville P. 2004 User Experience Design [honeycomb model]. http://www.semanticstudios.com/publications/semantics/000029.php .

Rosenbaum S. 2010 Improving the user experience of evidence: A design approach to evidence-informed health care. PhD thesis. Oslo, Norway: The Oslo School of Architecture and Design.

Download references

Acknowledgements

We gratefully acknowledge the representatives from the National Ministries of Health in Nigeria, Malawi and South Africa for their support and partnership. We would also like to thank the appointed Steering Committees who have been providing input for the research project and guiding the prioritization of topics. We would also like to thank Joy Oliver and Michelle Galloway for their contribution an support of the project.

The GELA project is funded by EDCTP2 programme supported by the European Union (grant number RIA2020S-3303-GELA). The funding will cover all the activities for this Monitoring and Evaluation work package, including costs for personnel and publication of papers.

Author information

Authors and affiliations.

Health Systems Research Unit, South African Medical Research Council, Cape Town, South Africa

Tamara Kredo, Denny Mabetha, Bey-Marrié Schmidt & Simon Lewin

Division of Epidemiology and Biostatistics, Department of Global Health, Stellenbosch University, Cape Town, South Africa

Tamara Kredo, Anke Rohwer, Michael McCaul, Idriss Ibrahim Kallon, Taryn Young & Sara Cooper

Division of Clinical Pharmacology, Department of Medicine, Stellenbosch University, Cape Town, South Africa

Tamara Kredo

School of Public Health and Family Medicine, University of Cape Town, Cape Town, South Africa

Tamara Kredo & Sara Cooper

University of Calabar Teaching Hospital, Calabar, Nigeria

Emmanuel Effa

Kamuzu University of Health Science, Lilongwe, Malawi

Nyanyiwe Mbeye

School of Public Health, University of the Western Cape, Cape Town, South Africa

Bey-Marrié Schmidt

Western Norway University of Applied Sciences, Bergen, Norway

Susan Munabi-Babigumira & Claire Glenton

Norwegian University of Science and Technology, Trondheim, Norway

Simon Lewin

MAGIC Evidence Ecosystem Foundation, Oslo, Norway

Per Olav Vandvik

Department of Health Economics and Health Management, Institute for Health and Society, University of Oslo, Oslo, Norway

Cochrane South Africa, South African Medical Research Council, Cape Town, South Africa

Sara Cooper

You can also search for this author in PubMed   Google Scholar

Contributions

T.K., S.C., T.Y., S.L., C.G. and P.O.V. conceptualized the protocol idea and S.C. drafted the protocol with input from TK, D.M., A.R., B.M., M.M., I.I., C.G., T.Y., S.L. and P.O.V.; all authors approved the final version for submission for publication.

Corresponding author

Correspondence to Tamara Kredo .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval has been obtained in each partner country (South Africa, Malawi and Nigeria) from the respective Health Research Ethics Committees or Institutional Review Boards. Information about the project will be provided to, and consent obtained from, all participants completing the online surveys and interviews and all participants taking part in the meetings. The consent forms will make explicit the voluntary nature of participation, that there will be no negative consequences if they decide not to participate and in the case of the interviews and meetings observations will ask explicitly for permission for the interview or meeting to be recorded. The online surveys will ask participants to provide the last six numbers of their ID as a unique identifier to track their capacity development needs and progress throughout the project. To help protect their confidentiality, the information they provide will be private, deidentified and no names will be used. In addition, all cookies and IP address collectors will be disabled to ensure confidentiality. All interview and meeting recordings on the digital recorders will be destroyed following safe storage and transcription, and any identifying information will be redacted from all transcripts. All study data, including recordings, will be stored electronically using password-controlled software only accessible to key project members and project analysts. Reports of study findings will not identify individual participants. We do not anticipate any specific harms or serious risks to participants. However, there is a risk of breaches of confidentiality for participants who take part in guideline panel and steering group committee project meetings. At the start of all meetings, participants will be introduced to each other. The member names of these groups will not be anonymous as they will play an ongoing role in the GELA project. At the start of each meeting, we will discuss the importance of maintaining confidentiality by everyone. As part of guideline development processes, all guideline members will need to declare conflicts of interests and sign a confidentiality agreement. We will explain, however, that while the researchers undertake to maintain confidentiality, we cannot guarantee that other meeting participants will, and there is, thus, a risk of breaches of confidentiality. We will ensure participants are aware of this risk. Participants may also feel anxiety or distress expressing negative views about project activities. Where there is this potential and where participants identify concerns, we will reassure participants of the steps that will be taken to ensure confidentiality.

Consent for publication

Competing interests.

All authors declared no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kredo, T., Effa, E., Mbeye, N. et al. Evaluating the impact of the global evidence, local adaptation (GELA) project for enhancing evidence-informed guideline recommendations for newborn and young child health in three African countries: a mixed-methods protocol. Health Res Policy Sys 22 , 114 (2024). https://doi.org/10.1186/s12961-024-01189-5

Download citation

Received : 02 June 2023

Accepted : 20 July 2024

Published : 19 August 2024

DOI : https://doi.org/10.1186/s12961-024-01189-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Related diseases
• Newborn and child health
• Saharan Africa (SSA)
• Informed decision
• Capacity development
• Monitoring and evaluation;
• Research impact

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Scoping Review
• Open access
• Published: 09 August 2024

Disparities in hepatitis B virus healthcare service access among marginalised poor populations: a mixed-method systematic review

• Caixia Li 1 ,
• Dejina Thapa 2 ,
• Qian Mi 3 ,
• Yuanxiu Gao 1 &

Infectious Diseases of Poverty volume  13 , Article number:  58 ( 2024 ) Cite this article

234 Accesses

Metrics details

Marginalised poor populations, characterised by poverty and social exclusion, suffer disproportionately from hepatitis B virus (HBV) infections and encounter substantial disparities in access to healthcare. This has further exacerbated the global HBV burden and precluded progress towards HBV elimination. This mixed-method systematic review aimed to synthesise their utilisation and influencing factors in HBV healthcare services, including screening, vaccination, treatment, and linkage-to-care.

Eleven databases were searched from their inception to May 4, 2023. Quantitative and qualitative studies examining the factors influencing HBV healthcare access among marginalised poor populations were included. A meta-analysis was conducted to synthesise the pooled rates of HBV healthcare utilisation. The factors influencing utilisation were integrated and visualised using a health disparity research framework.

Twenty-one studies were included involving 13,171 marginalised poor individuals: sex workers, rural migrant workers, irregular immigrants, homeless adults, and underprivileged individuals. Their utilisation of HBV healthcare ranged from 1.5% to 27.5%. Meta-analysis showed that the pooled rate of at least one dose of the HBV vaccine barely reached 37% (95% confidence interval: 0.26‒0.49). Fifty-one influencing factors were identified, with sociocultural factors ( n = 19) being the most frequently reported, followed by behavioural ( n = 14) and healthcare system factors ( n = 11). Socio-cultural barriers included immigration status, prison history, illegal work, and HBV discrimination. Behavioural domain factors, including previous testing for sexually transmitted diseases, residential drug treatment, and problem-solving coping, facilitated HBV healthcare access, whereas hostility coping exerted negative influences. Healthcare system facilitators comprised HBV health literacy, beliefs, and physician recommendations, whereas barriers included service inaccessibility and insurance inadequacies. The biological and physical/built environments were the least studied domains, highlighting that geographical mobility, shelter capacity, and access to humanitarian health centres affect HBV healthcare for marginalised poor populations.

Conclusions

Marginalised poor populations encounter substantial disparities in accessing HBV healthcare, highlighting the need for a synergistic management approach, including deploying health education initiatives to debunk HBV misperceptions, developing integrated HBV management systems for continuous tracking, conducting tailored community outreach programmes, and establishing a human rights-based policy framework to guarantee the unfettered access of marginalised poor populations to essential HBV services.

The hepatitis B virus (HBV) is a hepatotropic DNA virus that affects nearly one-third of the global population [ 1 ]. HBV caused approximately 254 million chronic infections in 2022 [ 1 ], which was more than HIV, tuberculosis, or malaria combined [ 2 , 3 ]. The number of HBV-related deaths from HBV are projected to increase by 35% from 820,000 in 2019 to 1,109,500 by 2030 [ 4 ]. HBV persists as a substantial public health concern across global regions (e.g. Asia-Pacific and sub-Saharan Africa). The World Health Assembly has endorsed the goal of eliminating HBV, defined as a 65% reduction in mortality and a 90% decline in the incidence of hepatitis B between 2015 and 2030 [ 5 ]. To achieve this, at least 90% of patients with hepatitis B must be diagnosed, and 80% of eligible patients must be treated [ 5 ]. However, HBV elimination activities, from prevention to testing and treatment, receive insufficient attention and investment, and were only funded by 37% of countries by 2017 [ 6 ].

Health equity alongside HBV elimination is even more been neglected. HBV is unevenly distributed across societal strata, with a disproportionately higher prevalence in marginalised poor populations who experience poverty and social exclusion from economic, social, political, and cultural dimensions [ 7 ]. These groups include, but are not limited to, homeless people, disabled individuals, sanitation workers, commercial sex workers, rural-urban migrant workers, incarcerated individuals, and irregular migrants such as refugees and asylum seekers [ 8 ]. A recent meta-analysis reported an estimated HBV prevalence of 15% among sanitation workers from Asian, African, and South American regions [ 9 ], compared with approximately 4.1% in the general population globally [ 3 ]. A prevalence rate of 30.9% for HBV exposure and incidences 7 to 10 times higher for HBV prevalence have been estimated among homeless individuals from the United States [ 10 ]. The prevalence rates among sex workers were reported to be 9.2% [ 11 ] and as high as 13.6% to 60.8% in refugees from low- and middle-income countries, including Ethiopia, Thailand, and Pakistan [ 12 ]. Unstable living conditions, poor living standards, limited access to healthcare, and exposure during work (e.g. biological exposure during waste picking) create a permissive environment for HBV transmission among the marginalised poor [ 8 , 9 ].

HBV healthcare services, ranging from vaccination to screening, treatment, and linkage-to-care, are strikingly less accessible to marginalised poor populations. Only 16.7% to 38.7% of the marginalised population, including female sex workers [ 13 ], homeless individuals, and those incarcerated [ 14 ], exhibited a serological profile of HBV vaccination. More importantly, the asymptomatic nature of chronic HBV infection necessitates a reliance on screening to identify cases. However, HBV screening has been poorly utilised, with only 10.5% of those infected with HBV worldwide aware of their status, and a mere 2.2% receiving treatment in 2019 [ 1 ]. These figures are suspected to be even lower among the marginalised poor due to insufficient data capture and multiple access barriers to services, including low health literacy, competing life priorities (food, clothing, and shelter), and difficulties in accessing healthcare facilities (e.g. lack of insurance, long-distance travel, and fear of judgment by health professionals) [ 15 ]. Reports suggest that only 1.5% of underprivileged individuals living in shelters had completed HBV screening, resulting in substantial delays in HBV diagnosis and treatment [ 16 ]. As a result, marginalised poor populations suffer more complications and mortality from HBV infection compared to the general public [ 8 ].

Despite this, factors influencing HBV healthcare access among marginalised poor populations are scarcely represented and synthesised in previous research, precluding an in-depth understanding of health needs and further allocation of health resources towards HBV elimination among the population. Thus, this review was conducted to synthesise the evidence on HBV healthcare service utilisation and its influencing factors among marginalised poor populations. It was guided by the National Institute on Minority Health and Health Disparity (NIMHD) research framework, which employs an integrative approach to represent multifaceted levels (individual, interpersonal, community, and societal) and domains (biological, behavioural, physical/built environment, socio-cultural environment, and healthcare system) that collectively explain health disparities [ 17 ]. The multitude of factors will be synthesised and visualised in the NIMHD framework to inform tailored interventions, policy-making, and resource allocation towards the global HBV elimination goal.

A systematic review and meta-analysis were conducted. The protocol was registered with the International Prospective Register of Systematic Reviews (CRD42022381183).

Literature search

Following the PICOs framework, approximately 133 search terms (Supplementary material 1) were developed pertaining to population and outcomes of this review, including “marginalized poor”, “hepatitis B”, “screening”, “vaccination”, “linkage-to-care”, and “influencing factors”. Relevant synonyms (e.g. hard to reach) and medical subject-heading (MeSH) terms were identified by referencing previous literature on marginalised poor populations [ 18 ] and by conducting an initial search in MEDLINE via OvidSP.

Search terms were retrieved in the fields of “title”, “keywords”, and “abstract”. Truncations (*) and adjacency searchers (adj) were used to enhance search efficiency. The following 11 databases were searched from their inception to May 4, 2023: Embase, MEDLINE via Ovid, Ovid Emcare, Ovid Nursing Database, British Nursing Index, Ovid APA PsycInfo, Cochrane Library, CINAHL, ProQuest Health & Medicine Collection, Scopus, and China National Knowledge Internet. There were no restrictions on language or publication data. A medical librarian refined the search strategy. Detailed search records for each database are shown in Supplementary material 1.

Study screening

After removing duplicates using Covidence (Veritas Health Innovation, Melbourne, Australia), four researchers independently screened the titles and abstracts of the retrieved articles against the eligibility criteria. The full texts of potentially eligible articles were retrieved and scrutinised by pairs of researchers. Disagreements were resolved through discussion.

Inclusion and exclusion criteria

Studies were eligible if they reported factors influencing HBV healthcare access among marginalised poor populations. The ‘Participant-Intervention-Comparator-Outcomes-Study design’ (PICOs) framework [ 19 ] was followed to formulate the following eligibility criteria:

The marginalised poor, who experience poverty and social exclusion across economic, social, political, and cultural dimensions, were eligible [ 7 ]. Focusing on the specific attributes of marginalisation—low-skill levels, low-socioeconomic status, and disability—this review included the following groups of marginalised poor individuals aged ≥ 18 years: (1) homeless adults; (2) migrant workers, including migrant domestic workers, migrant farmworkers, and migrant construction workers; (3) individuals with low socioeconomic status, such as farmers, construction workers, sanitation workers, the unemployed, and those living in poverty; and (4) individuals with disabilities. However, other marginalised populations, including men who have sex with men, were outside the scope of this review, and only included migrants with low socioeconomic status. Studies that included both the marginalised poor and the general population were eligible only if a subgroup analysis of the former was performed.

Studies should report factors influencing poor marginalised populations engagement in HBV vaccination, screening, treatment, and linkage-to-care, which refers to the process of referring patients with hepatitis B to medical care, ensuring that they receive directed care and treatment and are monitored regularly [ 20 ].

Study design

Quantitative, qualitative, and mixed-methods studies were eligible if they met the aforementioned criteria. Reviews, conference abstracts, editorials, guidelines, and letters were excluded.

Data extraction

A standard data extraction form was used to extract study data, focusing on factors influencing HBV healthcare access among marginalised poor populations. Odds ratios ( OR s), 95% confidence intervals ( CI s), and P values were extracted whenever possible to identify significant influencing factors. Insignificant factors were also extracted for study comparison. Qualitative data were summarised narratively. The extracted data were checked and validated by other researchers.

Critical appraisal of methodological quality

The Mixed Methods Appraisal Tool Version 2018 was used to assess the methodological quality of the included studies [ 21 ]. Two screening items assessing the clarity of the research questions and their coherence with the collected data were applied to all included studies. Five further questions were appraised based on study types. For quantitative descriptive studies, these five questions assessed the sampling strategy, representativeness of study samples, nonresponse bias, measurements, and statistical methods. For qualitative studies, the appropriateness of the qualitative approach, qualitative data collection methods, data analysis, interpretation, and coherence between them were assessed. Each item was rated as Yes, No, or Cannot Tell. One point was given for the Yes rating and zero for the other ratings. Total scores were converted into percentage scores. Three authors independently conducted quality appraisals, and any discrepancies in ratings were resolved through discussion.

Data synthesis

Factors influencing HBV healthcare access among marginalised poor populations were integrated based on the NIMHD framework into five domains: biological, physical/built environment, behavioural, socio-cultural environment, and healthcare system. Within each domain, the influence from the individual to interpersonal, community, and societal levels was categorised and analysed whenever applicable. The STATA 18.0 (StataCorp LLC, College Station, Texas, USA) was used to conduct meta-analyses to synthesise the rate of HBV healthcare utilisation and generate the combined effects of the influencing factors. Pooled rates, OR , and 95% CI were calculated, with the significance level set at P < 0.05. Heterogeneity was evaluated using Cochrane’s Q test, with P < 0.1 indicating significant heterogeneity [ 22 ]. When heterogeneity was statistically significant, a random-effects model using the DerSimonian-Laird method was used; otherwise, a fixed-effects model was used.

Search results

The database search identified 17,172 articles (Fig. 1 ). A total of 6712 duplicates were removed, and 10,216 articles were excluded after title and abstract screening. Among the 242 full-text articles retrieved, 223 were excluded mainly because they did not entirely focus on marginalised poor populations or report factors influencing HBV healthcare utilisation. Two additional studies were included after screening the references. Ultimately, 21 studies were included in the analysis [ 16 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ].

The Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) diagram

Study characteristics

Nineteen quantitative descriptive studies and two qualitative studies published between 2007 and 2021 (Table 1 ) were included. More than half of these studies were conducted in the United States of America ( n = 7), Brazil ( n = 3), China ( n = 2), and France ( n = 2). The marginalised poor populations in these countries included: (1) sex workers ( n = 6); (2) rural migrant workers ( n = 2); (3) irregular, low income, and underserved immigrants ( n = 5); (4) homeless adults, parolees, and prisoners ( n = 4); and (5) underprivileged and low socio-economic status individuals ( n = 4), such as collectors of recyclable waste and roadside barbers.

Critical appraisal

With critical appraisal ratings ranging from 43% to 100%, all included quantitative ( n = 19) and qualitative ( n = 2) studies attained moderate-to-excellent methodological quality (Supplementary material 2).

The research questions were clearly stated in all the included quantitative studies, and the collected data were adequate to address the study objectives. However, the sampling strategy in five included studies may have been inappropriate to address the research question or recruit a representative sample, as convenience sampling was applied [ 37 , 39 , 41 ] or relevant information (e.g. participant recruitment and inclusion criteria) was not clearly stated [ 30 , 35 ]. Seven studies either collected self-reported HBV vaccination status [ 23 , 28 , 33 , 34 , 42 ] or did not report details regarding the reliability or validity of the applied measurements [ 31 , 37 ]. In addition, one study had a high risk of nonresponse bias (44%) [ 26 ] and six studies did not specify the number of nonrespondents [ 28 , 34 , 35 , 37 , 38 , 40 ]. In contrast, most studies had applied appropriate statistical methods, such as multivariate models, to analyse the factors influencing HBV care among the marginalised poor.

Regarding qualitative studies, one was rated as having excellent methodological quality [ 25 ] while the other had a quality assessment score of 57%, since findings were not consolidated by participant quotes, and a lack of coherence existed between data collection, analysis, and interpretations [ 36 ].

HBV healthcare utilisation among marginalised poor populations

Thirteen studies reported rates of at least one HBV vaccination dose among marginalised poor populations. A meta-analysis of the thirteen studies showed a 37% pooled uptake rate (95% CI : 0.26‒0.49, P < 0.001, I 2 = 99.3%) [ 23 , 24 , 27 , 28 , 30 , 33 , 34 , 35 , 37 , 38 , 39 , 41 , 42 ] (Fig. 2 ). Sensitivity analysis showed that the pooled rates of at least one HBV vaccination dose were not affected by the removal of any individual study (Supplementary material 2). Three studies reported HBV screening rates ranging from 1.5% to 27.5% among immigrants with low income [ 26 ], the underprivileged [ 16 ], and poor populations [ 40 ]. The rates of linkage-to-care and HBV treatment after HBV diagnoses among immigrants with low income were 0% [ 30 ] and 15.9% [ 26 ], respectively.

Forest plot of the pooled at least one-dose hepatitis B virus (HBV) vaccination rates among marginalize poor populations, applying a random-effect model

Factors influencing access to HBV healthcare services among marginalised poor populations

A total of 51 influencing factors were identified and mapped onto the NIMHD research framework (Fig. 3 ). The socio-cultural environment domain influencing factors were the most commonly reported ( n = 19), followed by behavioural ( n = 14) and healthcare system domains ( n = 11). Each included study reported individual-level influencing factors. The details are interpreted in Fig. 3 .

Factors influencing hepatitis B virus (HBV) healthcare access among marginalised poor populations mapped onto the National Institute on Minority Health and Disparities Research Framework. Note : a Immigrants with low income, migrants residing in centres for refugees and asylum seekers, irregular migrants (mainly asylum seekers), minority adults with low income, Vietnamese nail salon workers from underserved communities; b Underprivileged people living in shelters, roadside barbers and clients, collectors of recyclable waste, underserved, and residing below poverty levels; c Factors included in the National Institute on Minority Health and Health Disparity research framework but were not reported in the included studies of this review

Biological domain

Regarding biological influencing factors, perceived poor health status [ 31 , 38 ] and a history of sexually transmitted diseases, including syphilis [ 35 ] and human immunodeficiency virus [ 26 , 34 ], were associated with HBV vaccine completion and HBV treatment receipt among marginalised poor populations, including homeless adults, minority adults with low income, and female sex workers. Poor health status and experience of disease may increase awareness of health issues and care-seeking, including HBV vaccination and treatment.

Physical/built environment domain

At the individual level, geographic mobility, including moving to a different region and engaging in seasonal work, disrupted the HBV care continuum and was the main barrier to HBV vaccination and treatment among marginalised poor populations (including female sex workers and irregular migrants) [ 23 , 28 , 36 ].

At the interpersonal level, it was noteworthy that female sex workers employed in erotic show houses and clubs presented higher odds of not completing the HBV vaccination schedule compared to those working on streets, massage parlours, brothels, bars, or squares [ 24 , 28 ]. This association might be partially attributable to the higher earnings and turnover rates among women working in erotic show houses and clubs, which may induce additional barriers to vaccine completion, such as lack of time [ 28 ].

At the community and societal levels, high-capacity shelters (capacity > 160) increased the likelihood of underprivileged individuals completing HBV screening by approximately 6.4 times compared to those living in low-capacity shelters (capacity < 160) [ 16 ]. Nevertheless, those living in low-capacity shelters might have closer contact and higher risks of HBV contamination [ 16 ], necessitating additional interventions. In addition, humanitarian health centres also facilitate HBV treatment and management, especially for irregular migrants with HBV [ 36 ].

Behavioural domain

At the individual level, health behaviours and coping strategies were associated with HBV healthcare access among marginalised poor populations. Compared to inconsistent utilisation, consistent condom use was identified as a risk factor for non-completion of HBV vaccination doses among female sex workers and rural migrant workers [ 29 , 42 ]. This effect may be associated with the perception that condom use could protect against HBV, leading to a lower perceived need for vaccination. Conversely, previous testing of HBV or sexually transmitted diseases were consistently reported as a facilitator for HBV vaccination and treatment among immigrants with low income and female sex workers [ 26 , 34 ]. However, conflicting results regarding the effects of illicit drug use have also been reported. Two studies [ 33 , 34 ] found significantly higher odds of HBV vaccination among female sex workers and female prisoners with a history of injection drug use. Conversely, previous illicit or injection drugs use predicted non-completion of HBV vaccination in the combined effects in other three studies conducted among female sex workers and homeless male parolees ( OR = 1.64, 95% CI : 1.12‒2.40, I 2 = 40%, P = 0.189) (Fig. 4 a) [ 28 , 29 , 32 ].

Forest plot of the pooled odds ratio ( OR ) of influencing factors, including previous injection drug use ( a ), sex ( b ), education ( c ), and perceived HBV risk ( d ), on HBV vaccination among marginalised poor populations, applying fixed-effects models

Positive coping strategies, including problem-solving focus, were positively associated with completing the HBV vaccination series [ 35 , 38 ]. Conversely, negative emotions (e.g. hostility) were negatively associated with completion among minorities with low income, homeless adults, and male parolees [ 32 , 35 ].

At the interpersonal, community, and societal levels, female sex workers with less than one year of prostitution experience were less likely to complete HBV vaccination doses [ 24 ], suggesting a lower perceived risk of HBV infection and vaccination need compared to those with more years of prostitution [ 23 ]. Additionally, homeless adults with over 90 days of residential drug treatment were more likely to complete the HBV vaccination series than their counterparts who received less than 90 days of treatment [ 32 ]. By contrast, those attending self-help drug treatment programmes were more likely not to complete the vaccination programme compared to non-attendees [ 31 ]. Moreover, irregular migrants in Italy and France with a residence permit for medical reasons had access to free healthcare, facilitating regular HBV treatment [ 36 ].

Socio-cultural environment domain

Sociodemographic factors, including sex, race, immigration, and prison history, were common individual-level influencing factors. Among immigrants with low income and homeless adults, being male consistently predicted non-completion of the HBV vaccine series, with a combined OR value of 0.68 (95% CI : 0.54‒0.87, I 2 = 0%, P = 0.393) (Fig. 4 b) [ 26 , 31 ]. Furthermore, African Americans [ 31 , 38 ] and Asian/Pacific Islanders [ 32 ] were more likely to complete HBV vaccine doses than Caucasians. Moreover, immigration status [ 34 , 40 ], residing in host countries for less than six months [ 30 ], working illegally [ 36 ], and previous psychiatric hospitalisation [ 32 ] were negatively associated with HBV screening, lifetime vaccination, treatment, and follow-up care among underserved individuals, homeless parolees, female sex workers, and irregular migrants, such as refugees and asylum seekers. Additionally, no current or previous prison history was associated with HBV vaccination completion among homeless adults and female sex workers [ 33 , 38 ].

However, discrepancies existed between studies regarding certain demographic variables, including age, educational level, and HBV healthcare utilisation. Four studies showed that age was negatively associated with HBV screening and vaccination among rural migrant workers, collectors of recyclable waste, and underprivileged people [ 16 , 27 , 39 , 42 ]. This may be partially attributed to the national HBV vaccination plan for newborns and infants since the 1990s; younger individuals (e.g. aged 18 to 30 years) were more likely to fall within the required age for vaccination when the policy was introduced [ 42 ]. This contrasts with four other studies, in which a positive [ 29 , 31 , 41 ] or insignificant [ 38 ] correlation between age and HBV vaccination was observed. Similarly, a meta-analysis of three studies with OR values showed that a higher education level was associated with higher odds of rural migrant workers and collectors of recyclable waste undergoing HBV vaccination compared to those with a primary educational level or those with less than nine years of education ( OR = 1.64, 95% CI : 1.32‒2.03, I 2 = 0%, P = 0.544) (Fig. 4 c) [ 27 , 39 , 42 ]. However, two studies showed no correlation or a reverse correlation between educational level and HBV vaccine completion [ 24 , 38 ].

At interpersonal level, studies have yielded mixed results regarding whether social support was positively associated with HBV vaccination completion among homeless adults and parolees [ 32 , 38 ]. However, the vaccination status of family members and friends against HBV may have influenced rural migrant workers to receive the vaccination [ 42 ].

At the community and societal levels, community professional training programmes, public health campaigns, discrimination and stigma against HBV, and situational issues such as the prison realignment policy were also significant factors influencing HBV healthcare access. For example, a nail technician training programme for Vietnamese nail salon workers did not convey specific information regarding HBV and even spread misinformation by suggesting that HBV could be prevented by wearing masks [ 25 ]. In addition, owing to the knowledge gaps in HBV transmission and fears of infection, HBV discrimination and stigma (e.g. avoiding close contact with HBV-infected individuals) still exist within the community, posing another barrier to HBV care [ 25 ]. Vietnamese nail salon workers expressed that public health campaigns, including leaflets and brochures regarding HBV in nail salons, churches, and other Vietnamese communities, would be helpful in increasing HBV awareness and facilitate access to HBV vaccination [ 25 ]. Finally, among homeless male parolees, those released after the prison realignment policy were approximately 2.21 times more likely not to complete HBV vaccination compared to those released before realignment [ 32 ].

Healthcare system

At the individual level, health literacy and beliefs regarding HBV were associated with HBV healthcare utilisation among marginalised poor populations. Limited health literacy, including not having heard of HBV and misperceptions about its transmission, symptoms, and prevention strategies [ 23 , 25 ], and unawareness of the HBV vaccine, its costs, and service access [ 42 ] precluded Vietnamese nail salon workers, rural migrant workers, female sex workers, roadside barbers, and their clients from seeking HBV vaccination. Conversely, higher levels of HBV knowledge (including knowledge of HBV transmission, vaccination, and screening tests) increased the likelihood of rural migrant workers [ 42 ] and collectors of recyclable waste [ 39 ] undergoing HBV vaccination by up to 3 times.

Regarding health beliefs, a meta-analysis of two studies [ 27 , 42 ] showed that perceived HBV risk and vulnerability significantly increased the odds of rural migrant workers undergoing HBV vaccination ( OR = 1.40, 95% CI : 1.22‒1.60, I 2 = 0%, P = 0.980) (Fig. 4 d). The perceived efficacy [ 27 ] of the HBV vaccine was also associated with HBV screening and vaccination among immigrants with low income and rural migrant workers. In contrast, fear of detecting HBV, the perception of being healthy if asymptomatic [ 25 ], and HBV vaccine distrust [ 42 ] precluded marginalised poor populations from seeking HBV screening, vaccination, and care. However, the perceived severity of HBV was not significantly associated with HBV vaccination [ 27 ] and conflicting results were found regarding whether self-efficacy was positively associated with HBV vaccination completion among rural migrant workers and homeless adults [ 27 , 38 ].

At the interpersonal level, physician recommendations were essential for Vietnamese nail salon workers and irregular migrants to undergo HBV screening [ 25 , 36 ]. Otherwise, they might assume that a normal blood test would include HBV testing and perceive that they are not infected if the results were normal [ 25 ]. Moreover, receiving information about HBV vaccination programmes from healthcare professionals increased the odds of sex workers receiving at least one HBV vaccine dose by up to 4.27 times [ 23 ].

At the community level, inaccessibility [ 42 ] or inconvenience of HBV care services, including no free time and busy working hours [ 23 , 25 , 28 ], were negatively associated with HBV vaccination and care. According to Baars et al. [ 23 ], fear of needles and forgetting appointments also discouraged female sex workers from undergoing HBV vaccination [ 23 ]. Conversely, more visits to gastroenterology clinics and prior liver biopsy procedures increased the odds of immigrants with low income undergoing HBV screening and treatment by 2.6 and 5.4 times, respectively [ 26 ].

At the societal level, the cost of HBV vaccine, high out-of-pocket expenses, and lack of insurance were the main barriers to HBV vaccination and care among Vietnamese nail salon workers [ 25 ], rural migrant workers [ 42 ], and roadside barbers and their clients [ 37 ].

This mixed-method systematic review represents the comprehensive synthesis of HBV healthcare utilisation and its influencing factors among marginalised poor populations. The synthesis of 21 studies highlights a situation in which the pooled rate of HBV vaccination is merely 37%, and the rates for HBV screening, treatment, and linkage-to-care are less than 27.5%. Guided by the NIMHD research framework, 51 influencing factors were identified across biological (e.g. self-rated health status), physical/built environment (e.g. geographic mobility), behavioural (e.g. hostile coping strategy), socio-cultural environment (e.g. immigration, discrimination, and stigma), and healthcare system (e.g. health literacy and beliefs about HBV, availability, and accessibility of services) domains. These insights could inform the development of health education, HBV tracking and management systems, tailored community outreach programmes, and human rights-based policy frameworks to improve HBV healthcare access, ultimately paving the way for HBV elimination among marginalised poor populations.

The utilisation of HBV healthcare services among marginalised poor populations has been inadequately documented in the literature, with only 21 relevant studies included in this review. Reaching these populations is challenging due to logistical constraints, socioeconomic instability, and the lack of a robust healthcare infrastructure, all of which significantly impede systematic data collection and reporting [ 8 ]. According to three included studies [ 16 , 26 , 40 ], the rates of HBV screening and treatment among these populations were exceedingly low, with both less than 27.5% and 15.9%, respectively. These figures fall alarmingly short of the World Health Organization’s targets for HBV elimination, which aim for 90% diagnosis and 80% treatment coverage by 2030 [ 5 ]. Moreover, this review found that the pooled HBV vaccination rate with at least one dose was only 37%. These findings are consistent with those of a previous systematic review that revealed that referral, follow-up, and initiation of care for infectious diseases (hepatitis C virus and human immunodeficiency virus) among marginalised poor populations were below 30% [ 43 ]. These data underscore the disparities in HBV protection, diagnosis, and subsequent linkage-to-care among marginalised poor populations, all which potentially exacerbate the prevalence of HBV infection.

In the physical/built environment domain, geographical mobility emerged as a major obstacle to HBV vaccination and treatment among marginalised poor populations, especially migrant workers, irregular migrants, and female sex workers. Consistent with previous reviews, geographical mobility precluded marginalised migrant labourers from accessing health and vaccination services, leaving them more vulnerable to infectious diseases [ 44 ]. These groups often move across regions because of their illegal immigration status, seasonal working, or financial constraints [ 36 ]. The transient nature of these populations, along with their unfamiliarity with health systems in new regions, might obstruct their access to healthcare services, impede the completion of the HBV vaccine series, and lead to discontinuity in HBV care [ 23 , 28 ]. Mobility also makes it difficult to track HBV vaccination, treatment, and care requirements. This highlights that healthcare delivery and policy reform should be tailored to the circumstances of mobility among marginalised poor populations.

Regarding the behavioural domain, this review highlights the influence of negative emotional coping strategies, specifically hostility, as a barrier to HBV vaccine uptake. Consistent with previous literature, hostility is common among marginalised poor populations and creates a significant obstacle to healthcare [ 45 ]. Individuals exhibiting hostility demonstrate traits of irritation, cynicism, and mistrust, making them resistant to conforming to societal norms and complying with healthcare instructions from perceived authoritative figures, thereby complicating efforts to promote HBV vaccination [ 45 , 46 ]. The coping strategy of hostility represents a multifaceted interplay of psychological, social, cultural, and policy dynamics that perpetuate negative emotions within marginalised poor populations [ 47 ]. Severe depression, societal stereotypes (e.g. associating poverty with laziness, uncleanliness, and criminality), and discriminatory policies aimed at marginalised groups exacerbate feelings of hostility and alienation [ 48 , 49 ]. Future studies should further identify and address the complex factors that contribute to hostility among marginalised poor populations. Holistic approaches and collaborative efforts are suggested to mitigate hostility and enhance HBV vaccination among marginalised poor populations.

In the socio-cultural environment domain, migration status significantly impacts the accessibility of HBV healthcare for marginalised poor populations. Consistent with previous research [ 15 ], immigration presents a dilemma for healthcare access, particularly among immigrant sex workers [ 34 ] and irregular migrants [ 36 ]. Illegal immigration status, absence of a residence permit, unauthorised employment, fear of authorities, and discrimination for their occupations (e.g. sex work) make their access to healthcare extremely difficult [ 50 , 51 ]. Additionally, the scarcity of healthcare personnel and limited health resources allocated for irregular immigrants may further exacerbate the deficit in HBV healthcare provision [ 52 ]. Although few countries (e.g. the Netherlands and Brazil) have launched HBV vaccination programmes accessible to migrant sex workers, access to such care remains hindered by structural obstacles, including mobility and difficulties with public transportation [ 34 ]. More importantly, disruptions in federal healthcare systems obstruct HBV treatment and care for irregular immigrants relocating within the country for work [ 36 ]. Consequently, the incidence of invisible migrants infected with HBV is heightened [ 53 ]. It is imperative for governments and healthcare systems to intensify their focus on marginalised immigrant groups, promote HBV screening and vaccination services upon their arrival in host countries, and implement targeted outreach programmes to address the complex challenges in HBV treatment and care faced by these vulnerable populations.

In the healthcare system domain, significant knowledge gaps exist regarding HBV among marginalised poor populations, affecting their health beliefs and impeding their engagement with HBV screening and vaccination services [ 23 , 25 , 39 , 42 ]. Consistent with previous research [ 54 ], many lack comprehension of HBV, including its symptoms, transmission modes, and the preventive benefits of HBV vaccination. Some individuals hold misperceptions, believing that HBV spreads through food and water consumption, or mosquito bites [ 25 ]. Such misperceptions can exacerbate fear and stigmatisation linked to HBV, leading to hesitancy in seeking screening services due to apprehensions about potential stigma following a positive diagnosis [ 55 ]. Additionally, some believe that HBV transmission occurs only within families, which diminishes their perceived risk and susceptibility to HBV infection, consequently decreasing their likelihood of seeking HBV screening and vaccination [ 25 ]. Consistent with the findings of this review, previous studies have also shown that marginalised poor populations (e.g. farmers and migrants with low income) exhibit a limited understanding of HBV symptoms and often neglect or delay seeking HBV screening or treatment, even when displaying overt HBV symptoms, such as scleral jaundice [ 56 ]. Furthermore, marginalised poor populations are unaware of the existence of the HBV vaccine, lack information on where to access vaccination services, and have doubts regarding its safety and efficacy [ 42 , 54 ]. Compounded by lower levels of formal education, the population encounters challenges in accessing, comprehending, and utilising information related to HBV prevention and control [ 57 ]. It is imperative to prioritise targeted HBV education and awareness campaigns and promote immunisation outreach initiatives among marginalised poor communities.

More importantly, marginalised poor populations usually have low income and uninsured positions (e.g. roadside barbers and collectors of recyclable waste), and healthcare costs, lack of insurance coverage, and high rates of out-of-pocket payments become major deterrents to HBV healthcare access [ 25 , 42 ]. This echoes the findings of previous studies that marginalised poor populations have higher uninsured rates and are inaccessible to affordable healthcare services [ 58 , 59 ]. They might struggle to obtain basic needs (e.g. food, water, and housing), let alone access non-emergent preventive healthcare measures, such as HBV screening and vaccination [ 56 , 59 ]. Additionally, some marginalised poor populations, including nail technicians, female sex workers, and rural migrant workers, face additional challenges due to long working hours, often extending to seven days a week, making timely scheduling of HBV healthcare difficult [ 23 , 28 ]. Disparities in healthcare service access may ultimately lead to delayed HBV diagnosis, poor treatment, and diminished health outcomes, which, in turn, increase healthcare costs and financial burdens among marginalised poor populations and their families. This perpetuates a vicious cycle that further impedes access to HBV healthcare services. A more equitable health system should be established to guarantee healthcare coverage, facilitate service access, and empower supporting organisations to reduce HBV disparities among marginalised, poor populations.

This review has several limitations. First, although there was no restriction on the language of publications, this review only searched English and Chinese language databases, potentially missing relevant studies in other languages. Second, this review did not search for gray literature, including news, policy statements, and discussion forums, which could have provided valuable insights into healthcare utilisation among marginalised poor populations. Third, heterogeneity was observed in the pooled analysis of the HBV vaccination rate, which could stem from variations in study design and populations. This may have decreased the certainty of the synthesised evidence. Fourth, HBV vaccination status was self-reported in some included studies [ 23 , 28 ], potentially introducing underestimation or overestimation and biasing the estimated HBV vaccination rate in this review. Finally, the findings of this review should be interpreted with caution, considering the specific contexts of the included studies. Variations in the availability of HBV vaccinations, population demographics, and intervention approaches among the included studies may have affected generalisability of the findings.

Engagement with HBV healthcare among marginalised poor populations is poorly documented. A substantial portion of this population remains undiagnosed and untreated, exacerbating health inequities towards HBV elimination. Future empirical and modelling research is warranted to capture the missing data. Multilevel strategies addressing the social determinants that hinder HBV healthcare access are also suggested to narrow these inequities.

At the individual level, tailored health education is necessary to dispel misconceptions regarding HBV infection, including its symptoms, transmission routes, and preventive and treatment measures. To accommodate the low educational level of marginalised poor populations, utilising digital media and other technologies to disseminate engaging and easy-to-understand content is suggested to effectively reach and encourage HBV prevention and treatment behaviours. At the healthcare system level, establishing integrated HBV management systems is essential to address the challenges posed by the high geographic mobility of marginalised poor populations. Strategies include implementing mobile health clinics, telemedicine services, and online HBV surveillance platforms that allow continuous tracking and management. Incorporating HBV healthcare services into commonly accessed healthcare settings by marginalised poor populations, such as shelter-based clinics, humanitarian health centres, and drug treatment facilities, can also enhance accessibility and continuity of care. At the community level, bolstering outreach programmes for HBV screening and vaccination in venues frequented by marginalised poor populations, such as nail salons, churches, and ethnic community centres, is essential. Efforts led by community leaders and supported by non-governmental organisations to reduce hostility and stigma surrounding HBV are critical for improving programme reception and adherence. At the policy level, a human rights-based framework is suggested, especially for irregular migrants, to ensure unfettered access to crucial HBV healthcare due to their illegal immigration status. Policy adjustments should also address the right to health insurance, provide complementary HBV screening and vaccination for special groups (such as sex workers and individuals from HBV-prevalent regions), and ensure the availability of ongoing antiviral treatment.

This systematic review, employing a mixed-method methodology, comprehensively synthesised evidence regarding HBV healthcare utilisation and its influencing factors among marginalised poor populations. It identified significant service access gaps, with rates of HBV screening, treatment, and linkage-to-care alarmingly low among vulnerable populations, ranging from 1.5% to 27.5%. The pooled rate of at least one HBV vaccination dose barely reached 37%. Through the lens of the NIMHD research framework, this review identified 51 influencing factors dispersed across multiple domains: biological (e.g. self-rated health status and sexually transmitted diseases), physical/built environment (e.g. distance to health services and geographic mobility), behavioural (e.g. condom usage and hostile coping strategies), socio-cultural environment (e.g. immigration, discrimination, and stigma), and healthcare system (e.g. health literacy and beliefs about HBV, availability and accessibility of services) domains were identified to influence the HBV healthcare.

To facilitate HBV elimination among vulnerable populations, access to HBV healthcare services—from screening to vaccination, diagnosis, treatment, and follow-up care—should be enhanced. The findings of this review suggest a synergistic approach to counteracting these barriers. This would involve health education initiatives aimed at debunking HBV misperceptions, establishing integrated HBV management systems for continuous tracking and care, tailored outreach programmes geared towards improving screening and vaccination rates, and incorporating a human rights-based policy framework to guarantee unfettered access to essential HBV healthcare for marginalised poor populations.

Availability of data and materials

Data availability is not applicable to this review as the data used were synthesised from previous studies.

Abbreviations

Confidence intervals

• Hepatitis B virus

National Institute on Minority Health and Health Disparity

Odds ratios

Participant-Intervention-Comparator-Outcomes-Study design

Preferred Reporting Items for Systematic Reviews and Meta-analyses

World Health Organization. Hepatitis B 2024 Key fact. 2024. https://www.who.int/news-room/fact-sheets/detail/hepatitis-b . Accessed 13 Apr 2024.

World Health Organization. Immunization coverage. 2023. https://www.who.int/news-room/fact-sheets/detail/immunization-coverage . Accessed 13 Apr 2024.

GBD 2019 Hepatitis B Collaborators. Global, regional, and national burden of hepatitis B, 1990-2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Gastroenterol Hepatol. 2022;7:796–829.

Article   Google Scholar  

Hsu YC, Huang DQ, Nguyen MH. Global burden of hepatitis B virus: current status, missed opportunities and a call for action. Nat Rev Gastroenterol Hepatol. 2023;20:524–37.

Article   PubMed   Google Scholar  

World Health Organization. Combating hepatitis B and C to reach elimination by 2030. 2016. https://www.who.int/publications/i/item/combating-hepatitis-b-and-c-to-reach-elimination-by-2030 . Accessed 13 Apr 2024.

Matthews PC, Jack K, Wang S, Abbott J, Bryce K, Cheng B, et al. A call for advocacy and patient voice to eliminate hepatitis B virus infection. Lancet Gastroenterol Hepatol. 2022;7:282–5.

Zahra K, Zafar T. Marginality as a root cause of urban poverty: a case study of Punjab. Pak Dev Rev. 2015;54:629–48.

Google Scholar  

Martyn E, Eisen S, Longley N, Harris P, Surey J, Norman J, et al. The forgotten people: Hepatitis B virus (HBV) infection as a priority for the inclusion health agenda. Elife. 2023;12:e81070.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Souza-Silva G, Zolnikov TR, Ortolani PL, Cruvinel VRN, Dias SM, Mol MPG. Hepatitis B and C prevalence in waste pickers: a global meta-analysis. J Public Health (Oxf). 2022;44:761–9.

Khouzam N, Gelberg L, Guo R, Tseng CH, Bhattacharya D. Opiate dependence: a risk factor for hepatitis B virus exposure in homeless adults. Fam Community Health. 2020;43:161–9.

Daka D, Hailemeskel G, Fenta DA. Seroprevalence of hepatitis B virus and associated factors among female sex workers using respondent-driven sampling in Hawassa city. Ethiopia Infect Drug Resist. 2021;14:4301–11.

Lee C, Emeto TI, Walsh N. Prevalence of hepatitis B virus amongst refugees, asylum seekers and internally displaced persons in low- and middle-income countries: A systematic review. J Viral Hepat. 2023;30:4–18.

Article   PubMed   CAS   Google Scholar  

Hibbert M, Simmons R, Ratna N, Mandal S, Sabin C, Desai M, et al. Retrospective cohort study assessing coverage, uptake and associations with hepatitis B vaccination among females who engage in sex work attending sexual health services in England between 2015 and 2019. Sex Transm Infect. 2023;99:497–501.

Ly TDA, Castaneda S, Hoang VT, Dao TL, Gautret P. Vaccine-preventable diseases other than tuberculosis, and homelessness: a scoping review of the published literature, 1980 to 2020. Vaccine. 2021;39:1205–24.

Chan DNS, Li C, Law BMH, Xu B, Kwok C. Factors influencing the utilisation of mammography among ethnic minorities: a framework-driven systematic review and meta-analysis. J Immigr Minor Health. 2024;26:569–95.

Sahajian F, Bailly F, Vanhems P, Fantino B, Vannier-Nitenberg C, Fabry J, et al. A randomized trial of viral hepatitis prevention among underprivileged people in the Lyon area of France. J Public Health (Oxf). 2011;33:182–92.

Morgan BL, Stern MC, Pérez-Stable EJ, Hooper MW, Fejerman L. Adding a One Health approach to a research framework for minority health and health disparities. Elife. 2022;11:e76461.

Article   PubMed   PubMed Central   Google Scholar  

Cheraghi-Sohi S, Panagioti M, Daker-White G, Giles S, Riste L, Kirk S, et al. Patient safety in marginalised groups: a narrative scoping review. Int J Equity Health. 2020;19:26.

Mehrdad A-B, Ali J. Population, intervention, comparison, outcomes and study (PICOS) design as a framework to formulate eligibility criteria in systematic reviews. Emerg Med J. 2020;37:387.

Abara WE, Qaseem A, Schillie S, McMahon BJ, Harris AM. Hepatitis B vaccination, screening, and kinkage to care: best practice advice from the American College of Physicians and the Centers for Disease Control and Prevention. Annals of Internal Medicine. 2017;167:794–804.

Hong QN, Pluye P, Fabregues S, Bartlett G, Boardman F, Cargo M, et al. Mixed methods appraisal tool (MMAT), version 2018 user guide. McGill University Department of Family Medicine. 2018.

Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557–60.

Baars JE, Boon BJ, Garretsen HF, van de Mheen D. Vaccination uptake and awareness of a free hepatitis B vaccination program among female commercial sex workers. Womens Health Issues. 2009;19:61–9.

Carneiro LM, Mousquer GJ, Pinheiro RS, Castro AR, França DD, Caetano KA, et al. Outreach hepatitis B vaccination of female sex workers in central-west Brazil: immunization status, compliance, and immune response. J Public Health Manag Pract. 2014;20:662–6.

Freeland C, Huynh T, Vu N, Nguyen T, Cohen C. Understanding knowledge and barriers related to hepatitis B for Vietnamese nail salon workers in the city of Philadelphia and some of its environs. J Community Health. 2021;46:502–8.

Jung CW, Tan J, Tan N, Kuo MN, Ashok A, Eells SJ, et al. Evidence for the insufficient evaluation and undertreatment of chronic hepatitis B infection in a predominantly low-income and immigrant population. J Gastroenterol Hepatol. 2010;25:369–75.

Liu R, Li Y, Wangen KR, Maitland E, Nicholas S, Wang J. Analysis of hepatitis B vaccination behavior and vaccination willingness among migrant workers from rural China based on protection motivation theory. Hum Vaccin Immunother. 2016;12:1155–63.

Magalhães RLB, Teles SA, Reis RK, Galvão MTG, Gir E. Low completion rate of hepatitis B vaccination in female sex workers. Rev Bras Enferm. 2017;70:489–94.

Mayanja Y, Abaasa A, Namale G, Asiki G, Price MA, Kamali A. Factors associated with vaccination completion and retention among HIV negative female sex workers enrolled in a simulated vaccine efficacy trial in Kampala. Uganda BMC Infect Dis. 2019;19:725.

Mazzitelli M, Greco G, Serapide F, Scaglione V, Morrone H, Marascio N, et al. Outcome of HBV screening and vaccination in a migrant population in southern Italy. Infez Med. 2021;29:236–41.

PubMed   Google Scholar  

Nyamathi A, Liu Y, Marfisee M, Shoptaw S, Gregerson P, Saab S, et al. Effects of a nurse-managed program on hepatitis A and B vaccine completion among homeless adults. Nurs Res. 2009;58:13–22.

Nyamathi A, Salem BE, Zhang S, Farabee D, Hall B, Khalilifard F, et al. Nursing case management, peer coaching, and hepatitis a and B vaccine completion among homeless men recently released on parole: randomized clinical trial. Nurs Res. 2015;64:177–89.

Plugge EH, Yudkin PL, Douglas N. Predictors of hepatitis B vaccination in women prisoners in two prisons in England. J Public Health (Oxf). 2007;29:429–33.

Ranjan A, Shannon K, Chettiar J, Braschel M, Ti L, Goldenberg S. Barriers and facilitators to hepatitis B vaccination among sex workers in Vancouver, Canada: implications for integrated HIV, STI, and viral hepatitis services. Int J Infect Dis. 2019;87:170–6.

Reynolds GL, Nguyen HH, Singh-Carlson S, Fisher DG, Odell A, Xandre P. Application of the extended health control belief model to predict hepatitis A and B vaccinations. Public Health Nurs. 2016;33:430–9.

Santilli C. Medical care, screening and regularization of sub-Saharan irregular migrants affected by hepatitis B in France and Italy. J Immigr Minor Health. 2018;20:668–73.

Shahid A, Nasim S, Memon AA. Insight and educational intervention concerning hepatitis among roadside barbers and their clients in Karachi. Pakistan J Infect Dev Ctries. 2013;7:125–9.

Stein JA, Nyamathi AM. Completion and subject loss within an intensive hepatitis vaccination intervention among homeless adults: the role of risk factors, demographics, and psychosocial variables. Health Psychol. 2010;29:317–23.

Weis-Torres S, Fitts SMF, Cardoso WM, Higa Junior MG, Lima LA, Bandeira LM, et al. High level of exposure to hepatitis B virus infection in a vulnerable population of a low endemic area: a challenge for vaccination coverage. Int J Infect Dis. 2020;90:46–52.

Wong RJ, Campbell B, Liu B, Baden R, Bhuket T. Sub-optimal testing and awareness of HCV and HBV among high risk individuals at an underserved safety-net hospital. J Community Health. 2018;43:65–9.

Wouters K, Leuridan E, Van Herck K, Van Ardenne N, Roelofs I, Mak R, et al. Compliance and immunogenicity of two hepatitis B vaccination schedules in sex workers in Belgium. Vaccine. 2007;25:1893–900.

Xiang H, Tang X, Xiao M, Gan L, Chu K, Li S, et al. Study on status and willingness towards hepatitis B vaccination among migrant workers in Chongqing, China: a cross-sectional study. Int J Environ Res Public Health. 2019;16:4046.

Saha R, Miller AP, Parriott A, Horvath H, Kahn JG, Malekinejad M. Viral blood-borne infections testing and linkage to care cascade among persons who experience homelessness in the United States: a systematic review and meta-analysis. BMC Public Health. 2022;22:1421.

Andreas DF, Zahra B, Leonidas GI, Kristine HO, Kai HP, Sally H. Improving the evidence on health inequities in migrant construction workers preparing for big sporting events. BMJ. 2021;374:n1615.

Nyamathi A, Salem B, Farabee D, Hall E, Zhang S, Khalilifard F, et al. Predictors of high level of hostility among homeless men on Parole. J Offender Rehabil. 2014;53:95–115.

Grassi L, Belvederi Murri M, Riba M, de Padova S, Bertelli T, Sabato S, et al. Hostility in cancer patients as an underexplored facet of distress. Psychooncology. 2021;30:493–503.

Williams L. Narratives of displacement and poverty: the intersections of policy and the shared experience of the everyday. Front Hum Dyn. 2023;5:2673–726.

Reilly J, Ho I, Williamson A. A systematic review of the effect of stigma on the health of people experiencing homelessness. Health Soc Care Community. 2022;30:2128–41.

Dobson R, Turnbull S. In or against the state? Hospitality and hostility in homelessness charities and deportation practice. Int J Law Context. 2022;18:25–40.

Raposo VL, Violante T. Access to health care by migrants with precarious status during a health crisis: some insights from Portugal. Hum Rights Rev. 2021;22:459–82.

Jiménez-Lasserrotte MDM, Granero-Molina J, Lardon Galindo V, Hernández Sola C, Ventura-Miranda MI, Hernández-Padilla JM, et al. Irregular migrants’ experiences of health disparities while living in informal settlements during the COVID-19 pandemic. J Adv Nurs. 2023;79:1868–81.

Asif Z, Kienzler H. Structural barriers to refugee, asylum seeker and undocumented migrant healthcare access. Perceptions of doctors of the world caseworkers in the UK. SSM - Mental Health. 2022;2:100088.

Razavi-Shearer D, Gamkrelidze I, Pan CQ, Razavi-Shearer K, Blach S, Estes C, et al. The impact of immigration on hepatitis B burden in the United States: a modelling study. Lancet Reg Health Am. 2023;22:100516.

PubMed   PubMed Central   Google Scholar  

Oladipo AT, Odeyemi K. Knowledge, attitude, and risk factors of hepatitis B among waste scavengers in Lagos. Nigeria J Clin Sci. 2021;18:191–7.

Cama E, Beadman M, Beadman K, Hopwood M, Treloar C. Health workers’ perspectives of hepatitis B-related stigma among Aboriginal and Torres Strait Islander people in New South Wales Australia. Harm Reduction Journal. 2023;20:116.

Li C, Lu X, Xiao J, Chan CWH. “We can bear it!” Unpacking barriers to hepatocellular carcinoma screening among patients with hepatitis B: a qualitative study. J Clin Nurs. 2022;31:3130–43.

Hyun S, Ko O, Kim S, Ventura WR. Sociocultural barriers to hepatitis B health literacy in an immigrant population: a focus group study in Korean Americans. BMC Public Health. 2021;21:404.

Robards F, Kang M, Steinbeck K, Hawke C, Jan S, Sanci L, et al. Health care equity and access for marginalised young people: a longitudinal qualitative study exploring health system navigation in Australia. Int J Equity Health. 2019;18:41.

Baker K, Adams J, Steel A. Experiences, perceptions and expectations of health services amongst marginalized populations in urban Australia: A meta-ethnographic review of the literature. Health Expect. 2022;25:2166–87.

Download references

Acknowledgements

The authors would like to acknowledge the medical librarian Dr. Kendy from the Chinese University of Hong Kong for her assistance in refining the search strategy of this mixed-method systematic review.

This review was funded by the Scientific Research Start-up Fee of the Eighth Affiliated Hospital, Sun Yat-sen University, China (GCCRCYJ072).

Author information

Authors and affiliations.

The Department of Nursing, The Eighth Affiliated Hospital, Sun Yat-sen University, Shenzhen, Guangdong Province, China

Caixia Li, Yuanxiu Gao & Xia Fu

Nepal Health Research Council, Kathmandu, Nepal

Dejina Thapa

The School of Nursing, Guangdong Pharmaceutical University, Guangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

CL and XF designed the study. CL, DT, QM, and YG performed the two-round study screening process. CL and YG extracted data from the included studies. CL and QM performed the study quality appraisal. CL analysed the data and wrote the manuscript. All authors read, revised, and approved the final manuscript.

Corresponding author

Correspondence to Xia Fu .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Supplementary Information

Supplementary material 1., supplementary material 2., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Li, C., Thapa, D., Mi, Q. et al. Disparities in hepatitis B virus healthcare service access among marginalised poor populations: a mixed-method systematic review. Infect Dis Poverty 13 , 58 (2024). https://doi.org/10.1186/s40249-024-01225-0

Download citation

Received : 20 April 2024

Accepted : 08 July 2024

Published : 09 August 2024

DOI : https://doi.org/10.1186/s40249-024-01225-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health disparity
• Marginalised poor
• Systematic review

Infectious Diseases of Poverty

ISSN: 2049-9957

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• JRSM Short Rep
• v.4(6); 2013 Jun

Using mixed methods in health research

Shema tariq.

1 School of Health Sciences, City University London, EC1A 7QN, London, UK

Jenny Woodman

2 MRC Centre of Epidemiology for Child Health, UCL Institute of Child Health, WC1N 1EH, London, UK

Mixed methods research is the use of quantitative and qualitative methods in a single study or series of studies. It is an emergent methodology which is increasingly used by health researchers, especially within health services research. There is a growing literature on the theory, design and critical appraisal of mixed methods research. However, there are few papers that summarize this methodological approach for health practitioners who wish to conduct or critically engage with mixed methods studies. The objective of this paper is to provide an accessible introduction to mixed methods for clinicians and researchers unfamiliar with this approach. We present a synthesis of key methodological literature on mixed methods research, with examples from our own work and that of others, to illustrate the practical applications of this approach within health research. We summarize definitions of mixed methods research, the value of this approach, key aspects of study design and analysis, and discuss the potential challenges of combining quantitative and qualitative methods and data. One of the key challenges within mixed methods research is the successful integration of quantitative and qualitative data during analysis and interpretation. However, the integration of different types of data can generate insights into a research question, resulting in enriched understanding of complex health research problems.

Introduction

Mixed methods research is the use of quantitative and qualitative methods in one study. Research is often dichotomized as quantitative or qualitative. Quantitative research, such as clinical trials or observational studies, generates numerical data. On the other hand qualitative approaches tend to generate non-numerical data, using methods such as semi-structured interviews, focus group discussions and participant observation. Historically, quantitative methods have dominated health research. However, qualitative methods have been increasingly accepted by the health research community in the past two decades, with a rise in publication of qualitative studies. 1 As the value of qualitative approaches has been recognized, there has been a growing interest in combining qualitative and quantitative methods. A recent review of health services research within England has shown an increase in the proportion of studies classified as mixed methods from 17% in the mid-1990s to 30% in the early 2000s. 2 In this paper, we present a synthesis of key literature on mixed methods research, with examples from our own work and that of others to illustrate the practical applications of this approach. This paper is aimed at health researchers and practitioners who are new to the field of mixed methods research and may only have experience of either quantitative or qualitative approaches and methodologies. We wish to provide these readers with an accessible introduction to the increasingly popular methodology of mixed methods research. We hope this will help readers to consider whether their research questions might best be answered by a mixed methods study design, and to engage critically with health research that uses this approach.

The authors each independently carried out a narrative literature review and met to discuss findings. Literature was identified via searches of PubMed, Google and Google Scholar, and hand-searches of the Journal of Mixed Methods Research, with relevant publications selected after discussion. An important consideration was that papers either had a methodological focus or contained a detailed description of their mixed methods design. For PubMed and Google searches, similar terms were used. For example, the PubMed strategy consisted of title and abstract searches for: ((mixed methods) OR ((mixed OR (qualitative AND quantitative)) AND methods)). We also drew upon recommendations from mixed methods conferences and seminars, and reference lists from key publications.

What is mixed methods research?

The most widely accepted definition of mixed methods research is research that ‘focuses on collecting, analysing, and mixing both quantitative and qualitative data in a single study or a series of studies’. 3 Central to the definition is the use of both quantitative and qualitative methods in one study (or a series of connected studies). Separate quantitative and qualitative studies addressing the same research question independently would not be considered ‘mixed methods’ as there would be no integration of approaches at the design, analysis or presentation stage. A recent innovation in mixed methods research is the mixed methods systematic review, which sets out to systematically appraise both quantitative and qualitative literature on a subject area and then synthesize the findings.

Why are mixed methods approaches used?

The underlying assumption of mixed methods research is that it can address some research questions more comprehensively than by using either quantitative or qualitative methods alone. 3 Questions that profit most from a mixed methods design tend to be broad and complex, with multiple facets that may each be best explored by quantitative or qualitative methods. See Boxes 1 and ​ and2 2 for examples from our own work.

Examples of authors’ mixed methods research – JW.

Examples of authors’ mixed methods research – ST.

Usually, quantitative research is associated with a positivist stance and a belief that reality that can be measured and observed objectively. Most commonly, it sets out to test an a priori hypothesis and is therefore conventionally described as ‘deductive’. Strengths of quantitative research include its procedures to minimize confounding and its potential to generate generalizable findings if based on samples that are both large enough and representative. It remains the dominant paradigm in health research. However, this deductive approach is less suited to generating hypotheses about how or why things are happening, or explaining complex social or cultural phenomena.

Qualitative research most often comes from an interpretive framework and is usually informed by the belief that there are multiple realities shaped by personal viewpoints, context and meaning. In-depth qualitative research aims to provide a rich description of views, beliefs and meaning. It also tends to acknowledge the role of researcher and context in shaping and producing the data. Qualitative approaches are described as ‘inductive’ as questions are often open-ended with the analysis allowing hypotheses to emerge from data. High-quality qualitative research can generate robust theory that is applicable to contexts outside of the study area in question, helping to guide practitioners and policy-makers. 8 However, for research that aims to directly impact on policy and practice, the findings of qualitative research can be limited by the small sample sizes that are necessary for in-depth exploratory work and the consequent lack of generalizabilty.

Mixed methods research therefore has the potential to harness the strengths and counterbalance the weaknesses of both approaches and can be especially powerful when addressing complex, multifaceted issues such as health services interventions 9 and living with chronic illness. 10

There are many reasons why researchers choose to combine quantitative and qualitative methods in a study. 11 , 12 We list some common reasons below, using a hypothetical research question about adolescents’ adherence to anticonvulsant medication to illustrate real world applications.

• Complementarity: Using data obtained by one method to illustrate results from another. An example of this would be a survey of adolescents with epilepsy demonstrating poor levels of adherence. Semi-structured interviews with a sub-group of those surveyed may allow us to explore barriers to adherence.
• Development: Using results from one method to develop or inform the use of the other method. A focus group conducted with a group of adolescents with epilepsy may identify mobile phone technology as a potentially important tool in adherence support. We could then develop a mobile phone ‘app’ that reminds patients to take their medication and conduct an intervention study to assess its impact on adherence levels.
• Initiation: Using results from different methods specifically to look for areas of incongruence in order to generate new insights. An illustration of this would be a study exploring the discrepancy between reported adherence in clinic consultations and actual medication adherence. A review of case notes may find adherence levels of over 90% in a clinic population; however, semi-structured interviews with peer researchers may reveal lower levels of adherence and barriers to open discussion with clinicians.
• Expansion: Setting out to examine different aspects of a research question, where each aspect warrants different methods. We may wish to conduct a study that explores adherence more broadly. A large-scale survey of adolescents with epilepsy would provide information on adherence levels and associations whilst interviews and focus groups may allow us to engage with individual experiences of chronic illness and medication in adolescence.
• Triangulation: Using data obtained by both methods to corroborate findings. For example, we could conduct a clinical study measuring drug levels in individuals and documenting self-reported adherence. Qualitative methods such as video diaries may confirm adherence levels.

To this list we would also add political commitment. That is to say, researchers may recognize, and wish to deploy, the strengths of quantitative research in producing generalizable results but may also be committed to representing the voice of participants in their work.

Whatever the reasons for mixing methods, it is important that authors present these explicitly as it allows us to assess if a mixed methods study design is appropriate for answering the research question. 3 , 13

How is mixed methods research conducted?

When embarking on a mixed methods research project it is important to consider:

• the methods that will be used;
• the priority of the methods;
• the sequence in which the methods are to be used.

A wide variety of methods exists by which to collect both quantitative and qualitative data. Both the research question and the data required will be the main determinants of the methods used. To a lesser extent, the choice of methods may be influenced by feasibility, the research team’s skills and experience and time constraints.

Priority of methods relates to the emphasis placed on each method in the study. For instance, the study may be predominantly a quantitative study with a small qualitative component, or vice versa. Alternatively, both quantitative and qualitative methods and data may have equal weighting. The emphasis given to each component of the study will be driven mainly by the research question, the skills of the research team and feasibility.

Finally, researchers must decide when each method is to be used in the study. For instance a team may choose to start with a quantitative phase followed by a qualitative phase, or vice versa. Some studies use both quantitative and qualitative methods concurrently. Again the choice of when to use each method is largely dependent on the research question.

The priority and sequence of mixing methods have been elaborated in a typology of mixed methods research models. See Table 1 for typology and specific examples.

Examples of studies using mixed methods.

How is data analysed in a mixed methods project?

The most important, and perhaps most difficult, aspect of mixed methods research is integrating the qualitative and quantitative data. One approach is to analyse the two data types separately and to then undertake a second stage of analysis where the data and findings from both studies are compared, contrasted and combined. 19 The quantitative and qualitative data are kept analytically distinct and are analysed using techniques usually associated with that type of data; for example, statistical techniques could be used to analyse survey data whilst thematic analysis may be used to analyse interview data. In this approach, the integrity of each data is preserved whilst also capitalizing on the potential for enhanced understanding from combining the two data and sets of findings.

Another approach to mixed methods data analysis is the integrative strategy. 20 Rather than keeping the datasets separate, one type of data may be transformed into another type. That is to say that qualitative data may be turned into quantitative data (‘quantitizing’) or quantitative data may be converted into qualitative data (‘qualitizing’). 21 The former is probably the most common method of this type of integrated analysis. Quantitative transformation is achieved by the numerical coding of qualitative data to create variables that may relate to themes or constructs, allowing statements such as ‘six of 10 participants spoke of the financial barriers to accessing health care’. These data can then be combined with the quantitative dataset and analysed together. Transforming quantitative data into qualitative data is less common. An example of this is the development of narrative psychological ‘types’ from numerical data obtained by questionnaires. 22

Potential challenges in conducting mixed methods research

Despite its considerable strengths as an approach, mixed methods research can present researchers with challenges. 23 , 24

Firstly, combining methodologies has sometimes been seen as problematic because of the view that quantitative and qualitative belong to separate and incompatible paradigms. In this context, paradigms are the set of practices and beliefs held by an academic community at a given point in time. 25 Researchers subscribing to this view argue that it is neither possible nor desirable to combine quantitative and qualitative methods in a study as they represent essentially different and conflicting ways of viewing the world and how we collect information about it. 8 Other researchers take a more pragmatic view, believing that concerns about the incommensurability of worldviews can be set aside if the combination of quantitative and qualitative methods addresses the research question effectively. This pragmatic view informs much applied mixed methods research in health services or policy. 8

Secondly, combining two methods in one study can be time consuming and requires experience and skills in both quantitative and qualitative methods. This can mean, in reality, that a mixed methods project requires a team rather than a lone researcher in order to conduct the study rigorously and within the specified time frame. However, it is important that a team comprising members from different disciplines work well together, rather than becoming compartmentalized. 26 We believe that a project leader with experience in both quantitative and qualitative methods can act as an important bridge in a mixed methods team.

Thirdly, achieving true integration of the different types of data can be difficult. We have suggested various analytic strategies above but this can be hard to achieve as it requires innovative thinking to move between different types of data and make meaningful links between them. It is therefore important to reflect on the results of a study and ask if your understanding has been enriched by the combination of different types of data. If this is not the case then integration may not have occurred sufficiently. 23

Finally, many researchers cite the difficulty in presenting the results of mixed methods study as a barrier to conducting this type of research. 23 Researchers may decide to present their quantitative and qualitative data separately for different audiences. This strategy may involve a decision to publish additional work focusing on the interpretations and conclusions which come from comparing and contrasting findings from the different data types. See Box 1 for an example of this type of publication strategy. Many journals in the medical sciences have a distinct methodological base and relatively restrictive word limits which may preclude the publication of complex, mixed methods studies. However, as the number of mixed methods studies increases in the health research literature we would expect researchers to feel more confident in the presentation of this type of work.

Many of the areas we explore in health are complex and multifaceted. Mixed methods research (combining quantitative and qualitative methods in one study) is an innovative and increasingly popular way of addressing these complexities. Although mixed methods research presents some challenges, in much the same way as every methodology does, this approach provides the research team with a wider range of tools at their disposal in order to answer a question. We believe that the production and integration of different types of data and the combination of skill sets in a team can generate insights into a research question, resulting in enriched understanding.

DECLARATIONS

Competing interests.

None declared

This work was funded by the Medical Research Council (MRC) [grant number: G0701648 to ST], and the MRC with the Economic and Social Research Council (ESRC) [grant number: G0800112 to JW]

Ethical approval

No ethical approval was required for this work

Contributorship

This work was conceived by both ST and JW who each carried out an independent literature review and collaborated on the structure and content of this report. ST wrote the manuscript with revisions and editing done by JW

Acknowledgements

We thank Professors Jonathan Elford and Ruth Gilbert for their comments on draft manuscripts

This article was submitted by the authors and peer reviewed by Geoffrey Harding

This paper is in the following e-collection/theme issue:

Published on 7.8.2024 in Vol 13 (2024)

Development and Evaluation of a Web-Based Platform for Personalized Educational and Professional Assistance for Dementia Caregivers: Proposal for a Mixed Methods Study

Authors of this article:

• Logan DuBose 1, 2 , MBA, MD   ; 
• Qiping Fan 3 , MS, DrPH   ; 
• Louis Fisher 3   ; 
• Minh-Nguyet Hoang 4 , MBA   ; 
• Diana Salha 1   ; 
• Shinduk Lee 5 , MSPH, DrPH   ; 
• Marcia G Ory 1 , MPH, PhD   ; 
• Tokunbo Falohun 2, 6 , MS  

1 School of Public Health, Texas A&M University, College Station, TX, United States

2 Olera lnc, Houston, TX, United States

3 Department of Public Health Sciences, Clemson University, Clemson, SC, United States

4 School of Medicine, Texas A&M University, College Station, TX, United States

5 College of Nursing, University of Utah, Salt Lake City, UT, United States

6 Department of Biomedical Engineering, Texas A&M University, College Station, TX, United States

Corresponding Author:

Qiping Fan, MS, DrPH

Department of Public Health Sciences

Clemson University

201 Epsilon Zeta Drive

Clemson, SC, 29634

United States

Phone: 1 864 656 3841

Email: [email protected]

Background: Alzheimer disease (AD) and AD-related dementia are prevalent concerns for aging populations. With a growing older adult population living in the United States, the number of people living with dementia is expected to grow, posing significant challenges for informal caregivers. The mental and physical burdens associated with caregiving highlight the importance of developing novel and effective resources to support caregivers. However, technology solutions designed to address their needs often face low adoption rates due to usability issues and a lack of contextual relevance. This study focuses on developing a web-based platform providing financial and legal planning information and education for dementia caregivers and evaluating the platform’s usability and adoptability.

Objective: The goal of this project is to create a web-based platform that connects caregivers with personalized and easily accessible resources. This project involves industrial, academic, and community partners and focuses on two primary aims: (1) developing a digital platform using a Dementia Care Personalization Algorithm and assessing feasibility in a pilot group of caregivers, and (2) evaluating the acceptability and usability of the digital platform across different racial or ethnic populations. This work will aid in the development of technology-based interventions to reduce caregiver burden.

Methods: The phase I study follows an iterative Design Thinking approach, involving at least 25 dementia caregivers as a user feedback panel to assess the platform’s functionality, aesthetics, information, and overall quality using the adapted Mobile Application Rating Scale. Phase II is a usability study with 300 dementia caregivers in Texas (100 African American, 100 Hispanic or Latinx, and 100 non-Hispanic White). Participants will use the digital platform for about 4 weeks and evaluate its usefulness and ease of use through the Technology Acceptance Survey.

Results: The study received funding from the National Institute on Aging on September 3, 2021. Ethical approval for phase I was obtained from the Texas A&M University Institutional Review Board on December 8, 2021, with data collection starting on January 1, 2022, and concluding on May 31, 2022. Phase I results were published on September 5, 2023, and April 17, 2024, respectively. On June 21, 2023, ethical approval for human subjects for phase II was granted, and participant recruitment began on July 1, 2023.

Conclusions: Upon completing these aims, we expect to deliver a widely accessible digital platform tailored to assist dementia caregivers with financial and legal challenges by connecting them to personalized, contextually relevant information and resources in Texas. If successful, we plan to work with caregiving organizations to scale and sustain the platform, addressing the needs of the growing population living with dementia.

International Registered Report Identifier (IRRID): DERR1-10.2196/64127

Introduction

Currently, 6.7 million Americans live with Alzheimer disease (AD) and AD-related dementia, a progressive and debilitating neurocognitive disease that leads to loss of memory, motor function, and other psychological symptoms [ 1 ]. With the growing aging population in the United States, the number of people living with dementia is projected to grow exponentially to 13.8 million by 2060 [ 1 ]. These trends indicate a growing public health crisis that requires multifaceted interventions. People living with dementia are primarily cared for by informal caregivers, typically spouses and adult children [ 2 , 3 ], with as many as 16 million Americans currently serving in this role [ 4 ]. These informal caregivers of people living with dementia often have a higher risk of developing depression, anxiety, social isolation, and physical problems due to the chronic stress and diverse burdens associated with caregiving [ 5 , 6 ]. In recent years, dementia caregivers are estimated to provide more than 18 billion hours annually with an estimated economic value of US $346 billion for their care services [ 1 ], and this informal care cost is projected to increase to US $2.2 trillion in 2060 [ 7 ].

A large part of the informal caregiver burden is related to various responsibilities in caregiving for people living with dementia, including management of the care recipient’s financial, legal, and estate-related challenges [ 6 , 8 ]. Without adequate support, caregivers often struggle to find or use proper resources, making even simple tasks burdensome. Furthermore, these primary family caregivers often must balance the challenges of caregiving with other personal responsibilities, such as employment and family obligations [ 8 , 9 ]. Considering the significant financial burden associated with caregiving, role strain is further exacerbated when caregiving requires a full-time commitment. Six out of 10 family caregivers of people with dementia have reported significant work impacts ranging from reduction of work hours to early retirement due to their caregiving obligations, which disrupts their wages and employment and depletes their savings [ 6 , 10 ]. The varied progressive nature of dementia further compounds the complexity of care for people with dementia, increasing the psychological, physical, and financial factors of the caregiving burden [ 1 , 11 ].

The variation in caregiver burdens related to sociodemographic characteristics contributes to greater complications. Burdens are often exacerbated in ethnic and minority populations, who generally have higher rates of dementia but lower access to caregiving services and information [ 12 , 13 ]. In addition, ethnic minorities can experience complications when communicating with care professionals due to cultural or linguistic barriers [ 13 ]. These barriers can disproportionately affect nonnative English speakers, increasing the informal caregiver burden due to the greater experienced inaccessibility in support systems within the American health care system. Unique stressors have also been demonstrated in gender or sexual minority caregivers, further emphasizing the diversity of caregiving experiences and challenges within the United States [ 14 ]. Literature has shown that considering unique barriers or challenges in diverse demographics when developing support interventions increases their effectiveness in reducing caregiver burden [ 15 ].

Technological Interventions and Artificial Intelligence–Driven Digital Health

There are many professional caregiver assistance organizations that provide services to informal caregivers, including aging care law, financial services, respite care, in-home care, older adults living, and medical services [ 16 ]. Digital interventions present a promising tool for assisting caregivers in identifying relevant professional caregiver assistance organizations services. A myriad of digital interventions have been developed to aid caregivers of people with dementia in identifying these services, ranging from web-based training, educational forums, caregiving support groups, and videoconferencing [ 17 - 19 ]. However, the adoption rate of these technologies remains low outside of pilot studies. This low adoption rate can be attributed to factors such as poor usability and accessibility, information complexity, funding limitations, and lack of contextualized relevance [ 8 , 20 - 24 ]. This issue is especially pronounced among minority populations, such as African American and Hispanic or Latinx caregivers, where cultural differences and caregiving challenges are often not considered in the development and evaluation of technology-based interventions [ 25 ].

Factors influencing the implementation and adoption of technology-based interventions by informal caregivers include the expected or perceived value by users, the features of the technology, the characteristics of the caregivers, and the condition of dementia [ 26 ]. According to the Technology Acceptance Model (TAM), perceived usefulness and ease of use are the 2 most important factors in determining how likely users are to adopt or reject new technologies [ 27 ]. The features of developed technologies, such as physical appearance, simplicity, and usability, are crucial for caregiver adoption [ 26 ]. In addition, the continued adaptation of technology over time, as dementia progresses and as caregiving needs evolve, is noted as important [ 26 ]. Family caregivers come from various racial and ethnic groups, generations, occupations, financial situations, and educational and cultural backgrounds [ 6 ], and their characteristics and specialized care needs are widely mentioned as influencing factors for technology adoption [ 26 ].

Artificial intelligence (AI) can offer various benefits to developing interventions that help caregivers better understand and use information in real time, leading to a more personalized and effective decision-making process [ 28 ]. AI has the potential to improve perceived usefulness and ease of use by personalizing and simplifying complex information. For instance, AI-driven applications can often offer user-friendly interfaces and navigation, reducing the learning curve for caregivers who may not be technologically savvy and enhancing user experience. Moreover, AI can be used to create and deliver personalized dementia care suggestions, meeting the evolving needs of people with dementia and their caregivers [ 29 ].

Development of a Caregiving Support Platform

While there is increasing interest in technology interventions for dementia caregiving, limited literature has explored their effectiveness outside of research settings. Early involvement of caregivers, as emphasized in community participatory research and dissemination and implementation science, can enhance intervention adoptability and sustainability [ 18 , 30 - 32 ]. Our study seeks to address the paucity of research by exploring the efficacy of delivering caregiving resources and services in a technology-driven setting that considers individualized and cultural issues, in order to provide personalized, relevant, timely interventions to caregivers of people living with dementia.

To address this gap, our team developed a website-based platform designed to provide personalized caregiver support, including identification of caregiving resources related to older adults living arrangements, financial services, and legal services, as well as education related to dementia management and caregiving. In our web-based care planning tool, the Olera.care platform, we use our Dementia Care Personalization Algorithm (DCPA) based on logic decision trees and geolocation to provide caregivers of people living with dementia with a tailored guide on the aging care, older adults living, financial, and legal aspects of dementia caregiving (Olera.care). Studies have shown that such caregiving assistance is particularly effective in reducing caregiver burden [ 8 ]. To provide a technological intervention that would reduce the caregiving burden and address the needs of caregivers of people living with dementia, we developed a study with 2 phases centered in a community-engaged approach [ 33 , 34 ].

In further development cycles, various AI elements will be incorporated into our digital platform to take advantage of the rapidly advancing technology to increase acceptance and adoption. These AI elements include large language models (LLMs), a novel class of AI, and personalized care planning agents specialized in social assistance functions and resource connection. LLMs are a type of AI program that can recognize and interpret vast amounts of human language, texts, and data. Fine-tuning of LLMs through aging care domain-specific training can vastly improve algorithm accuracy and provide the most up-to-date information that is relevant for each user, enabling caregivers to navigate and use preexisting resources and information databases more effectively. The LLM’s capacity is particularly valuable in navigating the complexity of dementia caregiving information, especially for individuals unfamiliar with financial and legal terminology, those unfamiliar with terminology related to aging services, or those who are uncertain about their specific needs. In addition, personalized care planning agents, capable of perceiving information and making decisions, can be integrated to guide users through various resources and instructions for navigating websites, significantly improving the user interface and user experience. These agents can be optimized using machine learning, specifically through reinforcement learning from human feedback that leverages domain expertise to train the AI to output more contextually relevant responses.

Evaluation Metrics and Framework

There are several metrics to evaluate the usability, ease of use, and acceptance of health care technological interventions, including the Mobile Application Rating Scale (MARS) and the TAM [ 35 - 37 ]. The MARS evaluates a digital product based on the functionality, design, information quality, engagement, and subjective quality of digital applications using a 5-point Likert scale survey [ 35 ]. The TAM evaluates the acceptability of a technology primarily based on its perceived usefulness and perceived ease of use among users [ 36 , 37 ]. Perceived usefulness refers to the degree to which a person believes that using a particular system can allow them to work more quickly, fulfill its intended purpose, increase productivity, enhance effectiveness, and make the job easier, with responses ranging from “strongly disagree” to “strongly agree” [ 38 ]. Similarly, perceived ease of use indicates the degree to which a person believes that the system is easy to learn, controllable, clear and understandable, flexible, easy to become skillful, and easy to use [ 38 ]. According to the TAM, perceived usefulness and ease of use are critical factors influencing the adoption or rejection of new technologies, and it has been used to determine the likelihood of dementia caregivers adopting digital technology interventions [ 39 ]; therefore, it is particularly suited for this evaluation study.

Study Objectives

This study has 2 phases to develop, evaluate, and refine the Olera.care web-based platform. Phase I aims to develop the Olera.care platform, identify the caregiving challenges and needs of informal caregivers, and pilot-test the performance of the platform among a group of dementia caregivers. Phase II aims to understand the perceived usefulness and ease of use of the Olera.care platform among 3 different sociodemographic groups of informal caregivers of people living with dementia: African American, Hispanic or Latinx, and non-Hispanic White American.

Study Overview

The study consists of 2 phases of platform development and evaluation ( Textbox 1 ). Phase I of the study involves identifying caregiving challenges and needs, developing the Olera.care web-based platform, and pilot-testing the usability of the platform in a group of dementia caregivers. Phase II aims to evaluate the Olera.care platform among 3 of the largest racial and ethnic groups in Texas and iteratively develop the Olera.care platform. Human subjects research was critiqued by the National Institutes of Health reviewers with key comments ( Multimedia Appendix 1 ) addressed including clarifying risks and benefits for human participants in informed consents, along with privacy protection protocols.

Phase I: Platform development and pilot test

Development of platform (Build Stage)

• Task 1: Compile care resources and educational materials
• Task 2: Build a logic decision tree for personalized recommendations
• Task 3: Design user questionnaire to collect user characteristics and develop prototypes
• Task 4: Develop a web-based application

Pilot testing of the platform in caregivers (n=30)

• Qualitative research: Identify caregiving challenges and needs for the platform using one-on-one interviews with caregivers
• Quantitative research: Collect user characteristics and preliminarily evaluate the platform using modified Mobile Application Rating Scale

Phase II: Iterative platform development and evaluation

Iterative development of the platform

• Expand care resources database outside of Texas
• Update educational materials of Alzheimer disease and Alzheimer disease–related dementia
• Integrate phase I feedback to platform features

Usability study across different racial or ethnic groups (n=300)

• Develop a comprehensive evaluation survey using the technology acceptance model
• Enroll and instruct caregivers to complete tasks:1. Generate a personalized caregiving checklist2. Identify and save 3 relevant resources to “favorite” list3. Register for “recommendation of the Day” and complete technology acceptance survey (TAS)4. Four-week interaction with the platform followed by a second completion of the TAS
• Evaluate perceived usefulness and ease of use of the platform across caregiver characteristics

Platform Development

The development of the platform (Olera.care) adopts a build-measure-learn framework to prioritize the needs and preferences of caregivers for people living with dementia, using the Design Thinking product design methodology [ 40 ]. This iterative development process consists of three major steps: (1) prototypes will be developed to address preidentified user needs [Build Stage], (2) product usability will be tested [Measure Stage], and (3) key lessons will be identified for changes in next product iteration [Learn Stage]. The platform uses user feedback and a self-improving DCPA to provide caregivers with tailored educational information on the legal, financial, and estate planning aspects of dementia caregiving, as well as details on relevant local care and support options. DCPA considers care recipients’ stage of disease, financial circumstances, and care preferences to create unique recommendations. As a result, when the platform is in use, information is presented in a personalized and user-friendly manner that enhances the experience of caregivers when searching online for Alzheimer disease and AD-related dementia (AD/ADRD) support and reduces time spent sorting through general search engine results. We are improving the DCPA algorithm by leveraging LLMs and personalized AI agents. These AI agents will enhance our ability to provide real-time support, guidance, and information to caregivers to find dementia care services. By learning from caregivers’ interactions with Olera.care, these AI agents will continuously improve in delivering personalized and contextually relevant assistance.

The platform is developed to be a free web-based service to assist caregivers in connecting them with educational and professional assistance. The educational materials ( Figures 1 and 2 ) are created in collaboration with subject matter experts from local Area Agencies on Aging and Texas A&M University Center for Community Health and Aging. In addition, the National Institute on Aging, the American Association of Retired Persons, and the Texas Alzheimer’s Research and Care Consortium offer detailed informational materials on the legal and financial considerations for our targeted caregiver groups, which we frequently reference. Information on local care (ie, non–medical home care, assisted living, and nursing home) options ( Figure 3 ) is gathered from public knowledge bases such as websites of Medicare and the websites of local Area Agencies on Aging (eg, Brazos Valley Area Agency on Aging, Harris County Area Agency on Aging, and Area Agency on Aging of the Capital Area). The primary objective of this step is to compile a comprehensive library of information on the functional, legal, financial, and estate planning aspects of AD/ADRD caregiving, along with a detailed list of care facilities in Texas.

The educational content gathered was organized after compilation. This process involved dividing the information into clear, manageable segments and classifying each segment into one of three categories: (1) legal, financial, or other services; (2) home care; and (3) senior communities. Once categorized, a logical decision tree was created to match each piece of information with specific combinations of caregiver characteristics provided by the user. The algorithm for this process was developed using Python (Python Software Foundation) due to its straightforward syntax, extensive open-source libraries, and capability for efficient data manipulation [ 41 ]. In the final website-based application, caregiver characteristics will serve as input criteria, including factors such as the dementia stage of the care recipient, the caregiver’s knowledge and skills, the availability of external resources, the care recipient’s insurance status, financial capacity for care, and family support presence. These criteria will be collected when users create their accounts for the first time, determining the personalized information and recommendations from each of the 3 categories that will be presented to the caregiver. The logic decision tree was ultimately converted into a user-friendly website-based platform for caregivers to access customized educational materials (eg, videos, articles) and relevant care options (eg, at-home care, older adults living arrangements, paying for care options, caregiver support). Google’s Firebase is employed for backend programming to enable seamless data synchronization, user authentication, and secure data management. The front-end user interface is built using Google’s Flutter software development kit, facilitating cross-platform development for Android, iOS, and web applications from a single codebase. This approach streamlines the development process and reduces long-term maintenance costs.

Phase I: Mixed Methods Study—Needs Assessment and Pilot Testing

The phase I study is designed in a sequential mixed methods approach to understand the caregiving challenges and needs of caregivers and assess the platform’s usability with a pilot panel of AD/ADRD caregivers. The mixed-methodological design aims to enhance the depth and richness of data interpretation. The qualitative research serves as a need assessment, providing comprehensive information on the caregiving needs and expectations for the digital platform. Subsequently, the quantitative research evaluates the platform’s usability and usefulness for caregiving and identifies areas for platform enhancements.

The qualitative part involves semistructured interviews with caregivers, focusing on financial and legal challenges related to caregiving, unmet needs in financial management and legal planning, and expectations for a caregiving support platform. These interviews are recorded and transcribed for qualitative data analysis using thematic analysis in a framework approach [ 42 ]. Participants were also invited to complete a survey including questions about sociodemographic and caregiving characteristics, as well as questions on the usage and preferences for older adult care services.

A panel of dementia caregivers was invited to test the prototype web-based application by performing specific tasks, such as searching for financial and legal information. The study team is responsible for monitoring task completion, recording the status and time taken, and taking observation notes. After the prototype test, participants are invited to complete a usability survey to rate the perceived usefulness, ease of use, and overall rating of the web-based platform in supporting their caregiving needs. The usability of the platform was assessed using a modified MARS, which evaluates functionality, design, information quality, and engagement on a 5-point Likert scale [ 35 ]. The development goal is a MARS score of 3.6 or higher [ 43 ] among the pilot group of caregivers. The collected information will enable the development team to learn from the user feedback and plan for continuous product enhancement.

Phase II: Usability Evaluation Study Among Diverse Racial and Ethnic Groups

The phase II study is a usability study that evaluates the acceptance of the platform among the 3 largest racial and ethnic groups in Texas (Black or African American, Hispanic or Latinx, and non-Hispanic Whites) from a diverse socioeconomic spectrum. Participants who are interested in the study will complete a short interest form consisting of 4 questions to provide contact information. Participants will then be contacted to complete an intake form via Qualtrics consisting of eligibility screening questions; an informed consent form; questions on their demographic, socioeconomic, and caregiving characteristics; and an assessment of care receipt’s dementia stage using a validated 8-item screening instrument, the Ascertain Dementia-8 [ 44 ]. Next, eligible participants can select either to explore the platform and complete the rest of the study on their own for greater flexibility or to schedule a guided session with 1 of the study staff over Zoom (Zoom Video Communications, Inc). With instructions from the study staff, participants will create an account on the Olera.care platform and generate a personalized caregiver checklist based on their needs by answering a few questions. After setting up their account, participants will explore and use the platform for 4 weeks. At the end of the 4 weeks, participants are invited to provide feedback by filling out an evaluation survey of the platform, the Technology Acceptance Survey, which was developed based on the TAM, a well-established framework for assessing the perceived usefulness and ease of use of information technologies [ 38 , 45 ].

Participant Eligibility and Recruitment

To be eligible for phase I, participants must meet the following criteria: (1) be the primary unpaid caregiver for a person living with dementia, (2) provide a minimum of 10 hours of care per week to a person living with dementia who has not been institutionalized, (3) be an adult child, spouse, or family member of the person living with dementia, (4) have concerns about or perceive the need for additional information on financial management and legal planning for caregiving in Texas, and (5) have access to a smartphone or computer with internet connectivity. Paid formal caregivers will be excluded from this study. For the phase II study, the study participants must meet an additional criterion: be of Caucasian, African American, or Latino or Hispanic descent. Study participants are recruited in collaboration with the Center for Community Health and Aging at Texas A&M University and the Brazos Valley Area Agency on Aging. Web-based advertisements, emails, in-person presentations, and network recruitments are used to recruit participants. Potential participants will be invited to complete an eligibility assessment survey, and eligible participants will be invited to enroll in the study.

Ethical Considerations

Human subject research approval (institutional review board [IRB] number: IRB2021-0943 D) was obtained from the Institutional Review Board at Texas A&M University, with phase I study approved on December 8, 2021, and phase II study approved on June 21, 2023. Electronic informed consents are obtained from study participants before study activities. Participants’ personal identifying information (eg, names, emails, phone numbers) was used solely for contacting purposes. We protected participants’ privacy and confidentiality by limiting access to IRB-approved team members, separating identifying information from deidentified study data, encrypting study information on Microsoft OneDrive, and deleting identifying information upon project completion. Participants in phase I received a US $25 e-gift card for completing both interviews and surveys, and phase II participants will receive up to US $50 e-gift card for completing the study.

Sample Size Calculations

The phase 2 research hypothesis is that there will be a difference in the perceived usefulness and ease of use between different racial and ethnic groups. For the sample size calculation, we assumed a minimum power of 80% and a type I error rate of 5%. Due to the limited research on the influence of racial or ethnic factors on caregiving technology acceptance, we estimated the necessary sample size across a range of effect sizes ( f ²=0.02-0.35). Our analysis model will adjust for caregivers’ sociodemographic characteristics, caregiving duration, and the dementia stage of the care recipient. Depending on the effect sizes, the required sample size varies from approximately 40 to 550 participants [ 46 ]. Previous studies have shown low retention rates among informal caregivers for individuals with AD/ADRD, around 50% [ 47 , 48 ], and even lower retention rates among Hispanic or Latinx caregivers [ 47 ]. By following recommended recruitment and retention strategies and drawing on the recruitment experience of Texas A&M Center for Community Health and Aging in various community and clinical projects, we estimate that a minimum total sample size of 300 for enrollment surveys (100 per group) with 150 for conclusion surveys (50 per group) is necessary. This minimum sample size of 150 caregivers is feasible and can detect a small effect size of 0.09.

Data Analysis

For phase I data, thematic analysis of qualitative interview transcript data was conducted using the framework method [ 42 , 49 ]. The framework method consists of several essential steps: transcribing interviews, familiarizing oneself with the interview material, coding, developing an analytical framework, applying this framework, charting data into the framework matrix, and interpreting the data [ 42 ]. In addition, descriptive analyses of quantitative survey responses will be conducted to describe the sociodemographic and caregiving characteristics of the participants, their usage and preference for care services, and their evaluations of the platform’s functionality, design, information quality, and engagement.

For phase II survey data, descriptive statistics (mean and SD or frequency and percentage) will be used to describe the characteristics of study participants and their perceived usefulness and ease of use of Olera.care digital platform. First, using the Technology Acceptance Survey data, separate multivariable regression models will be used to examine any differences in each key outcome (perceived usefulness and ease of use of Olera digital platform) by racial or ethnic characteristics and socioeconomic status (eg, education and income level). The regression model will be adjusted for known factors that influence technology adoption (eg, age, sex, and care recipients’ dementia stage).

The study received funding from the National Institute on Aging on September 3, 2021. Ethical approval for phase I was obtained from the Texas A&M University Institutional Review Board on December 8, 2021, with data collection starting on January 1, 2022, and concluding on May 31, 2022. Phase I results were published on September 5, 2023, and April 17, 2024, respectively. On June 21, 2023, ethical approval for human subjects for phase II was granted, and participant recruitment began on July 1, 2023, and is anticipated to end by December 31, 2024. We expect to publish our phase II results by June 30, 2025. The Olera.care platform has seen promising growth since its official launch in 2023, now with more than 350 daily logins and a steady increase in organic traffic from August 6, 2023, to June 1, 2024 ( Multimedia Appendix 2 ). In addition, our Facebook “Health Aging Community” has more than 11,000 followers by June 1, 2024. The goal is to expedite this organic growth of our website and social media platforms in a way that requires no paid advertisements and is sustained only by quality content on the website attracting visitors (organic traffic). Our goal is to reach more than 30,000 caregivers daily by June 30, 2027. This will be achieved through continued investment in quality content, strategic partnerships, and community engagement.

Platform Description and Functionality

The primary function of the Olera.care platform is to provide personalized recommendations for learning materials, including articles and videos, tailored to caregivers’ specific needs. The platform also offers recommended listings of local service providers across various domains such as legal and financial advisors (eg, older adults law attorneys, certified financial planners), home care providers, senior living options, and public support services such as Meals on Wheels and Area Agencies on Aging. This personalized approach aims to simplify the web-based research process for caregivers, providing a guided experience akin to expert advice.

Key features of the Olera.care ( Figure 4 ) include the following:

• Personalized recommendations : the DCPA customizes content based on the care recipient’s disease stage, financial circumstances, and care preferences. This ensures that caregivers receive relevant, contextually appropriate information and resources.
• Educational materials : The platform hosts a comprehensive library of educational content created in collaboration with experts from various organizations, including the National Institute on Aging, the American Association of Retired Persons, and the Alzheimer’s Association.
• Service provider listings : Caregivers can access a curated list of local care options, including non–medical home care, assisted living, and nursing homes, gathered from reliable public sources.
• User-friendly interface : developed using Ruby on Rails web application framework for a seamless cross-platform experience, the platform ensures ease of use across different devices.

Marketing and User Growth Strategy

Since its launch, the Olera platform has focused on building organic traffic through several key strategies:

Community Building

The “Healthy Aging Community,” a web-based space hosted on Facebook, for seniors and caregivers, has become a significant marketing channel, with more than 11,000 registered members. This community fosters engagement and word-of-mouth promotion, driving new users to the platform.

Content Marketing and Search Engine Optimization

The platform’s Educational Material and Caregiver Resource repository features more than 20,000 indexed pages of high-quality content ( Multimedia Appendix 3 ). This content attracts organic traffic from users searching for relevant caregiving information, supported by digital public relations efforts and outreach to journalists, bloggers, and podcast hosts.

Strategic Partnerships:

Collaborations with organizations such as the Alzheimer’s Association and Texas A&M Center for Community Health and Aging have expanded the platform’s reach. These partnerships are crucial for user growth, with plans to engage more caregivers through connections with local Area Agencies on Aging and older adult centers.

Promotion to Service Providers

Marketing efforts also target service providers in the older adult care industry. Promotions for new providers and participation in industry conferences help build a robust network of providers on the platform. When families like a provider, a lead can be generated for that organization when we forward their contact information. Weekly leads are directly related to the volume of organic visits we get from caregivers using the platform ( Multimedia Appendix 4 ).

Study Progress by June 30, 2024

Phase I study data collection started on January 1, 2022, and ended on May 31, 2022. For participants, 822 respondents filled out prescreening surveys and 150 (18.2%) of them were qualified. Approximately 20% (30/150) of the eligible respondents participated in the in-depth interviews and completed the survey. The preliminary results of the phase I study were disseminated in a few regional or national conferences (ie, 2024 annual meeting of American Academy of Health Behavior; 2024 Texas Alzheimer’s Research and Care Consortium Symposium; 2023 Healthy Aging and Dementia Research Symposium; and 2023 annual meeting of American Association for Geriatric Psychiatry), and formal results were published on September 5, 2023 [ 8 ] and April 17, 2024 [ 50 ].

Phase II study commenced on July 1, 2023, and we aim to complete recruitment and data collection by the end of December 31, 2024, and conduct analysis and report our phase II evaluation results by June 30, 2025. By thoroughly evaluating the Olera.care digital platform across diverse sociodemographic groups, we aim to ensure that the platform is both effective and user-friendly for a broad audience. This evaluation will provide critical insights into the specific needs and preferences of different caregiver populations, enabling us to tailor the platform more precisely.

Principal Findings

This study presents an iterative approach to developing, refining, and evaluating a novel caregiver assistive technology. Current literature regarding the efficacy and acceptance of predeveloped caregiver interventions, while able to provide context regarding caregiver needs and future directions, often overlooks user feedback during the developmental phase. Using a build-measure-learn approach, the development of the Olera.care platform incorporates caregiver opinions and thoughts throughout the process. In the phase I study, we used a mixed methods approach to obtain comprehensive feedback in both qualitative and quantitative measures and examine the quality of the developed platform using the validated MARS tool. In the phase II study, we evaluated the acceptance of the technology among 3 major racial and ethnic groups using the TAM framework. This enables a comparison of measurements in perceived ease of use and usefulness, similar to another study that used the TAM framework to evaluate the acceptability of a care coordination platform [ 51 ].

A significant difference between our product and recently developed caregiver platforms is the multifaceted nature of the Olera.care platform. While many caregiving assistive technologies aim to decrease caregiver burden and strain, they mainly focus on only 1 or 2 aspects of the caregiving experience, thus limiting their use. For example, while the end goal is the same, these technologies often target specific aspects such as caregiver management [ 51 - 54 ], clinical reasoning [ 55 ], caregiver education and training [ 56 , 57 ], or caregiver mental well-being [ 58 ]. Olera.care platform serves functions in all 4 of the identified components and thus is able to serve as a simplified hub, capable of assisting caregivers in all needed aspects. For these reasons, Olera.care is a more rounded and comprehensive intervention than existing platforms.

Strengths and Limitations

One of the key strengths of the study is the build-measure-learn process. This iterative approach ensures that caregiver feedback is incorporated throughout the development phase, leading to a more user-centric design. By continuously refining the platform based on user input, the Olera.care platform is better tailored to meet the actual needs and preferences of caregivers, enhancing its usability and effectiveness. Another significant strength is the mixed methods approach in phase I, which allows for the collection of both comprehensive qualitative data on caregiving needs and expectations, as well as quantitative measures of usability and usefulness using the validated tool, MARS. The majority of similar studies have evaluated the acceptance of an intervention using only qualitative measures as opposed to quantitative [ 52 , 53 , 55 ]. Other studies have demonstrated the use of a mixed methods approach and MARS in broadening the caregiver perspective [ 54 , 56 , 59 , 60 ]. While app quality can be assessed in a multitude of ways, it is important to consider the MARS during developmental testing to prevent poor quality, especially considering that the average caregiving app was found to be “of [minimal] acceptable quality” that is “likely to be insufficient to meet care partner needs” [ 61 ]. Furthermore, this study stands out by investigating the acceptance of the technology among diverse ethnic and racial groups of caregivers in phase II. It is important to note the necessity of a diverse study population, considering the aforementioned burden disparities in racial and ethnic groups. Limitations in similar studies often arise from a lack of generalizability and consideration of a broadly representative sample [ 56 , 57 ]. Often, the unintentional focus on non-Hispanic White populations materializes in a lack of culturally tailored interventions [ 62 ]. This study will help bridge this gap by focusing on the needs and feedback across racial or ethnic populations.

The study has several limitations typical to research in this field of study. Recruiting racially and ethnically diverse caregiver participants, specifically from the 3 largest groups in Texas: non-Hispanic White, Hispanic or Latinx, and African American individuals requires intense efforts. Despite significant efforts to collaborate with communities and local agencies for recruitment, achieving a truly representative sample remained challenging. In addition, we were unable to include other racial or ethnic groups such as Asian American, Pacific Islander, Indian American, and Multiracial American. This omission is noteworthy as it potentially excludes populations that could offer important insights. However, the decision to focus on the 3 major groups was based on examining differences across these groups and considering the feasibility of recruiting caregivers in Texas. In addition, while our sample was racially diverse, it may not have been socioeconomically diverse and representative of all residents in Texas and the United States. Therefore, it is important to compare our participants’ characteristics with Texas caregiver profiles and national caregiver profiles to determine the representativeness of our recruited sample. Despite measuring household income levels and self-reported financial situations, unmeasured factors may influence the acceptance and perceived usefulness of the platform. For instance, health literacy and technology literacy can significantly impact the access and use of web-based caregiving resources [ 63 , 64 ]. Another limitation is the lack of longitudinal testing to observe changes in perceived usefulness over time. This gap means we are unable to assess how perceived acceptance and ease of use might evolve as caregivers continue to use the platform and their caregiving responsibilities evolve. Longitudinal studies are essential to understanding the sustained impact and usability of interventions over time. This underscores the need for a longitudinal study design and a more comprehensive assessment of caregiver characteristics in our future studies. By addressing these limitations in future research, we can gain a deeper understanding of the long-term effectiveness and broad applicability of the platform across various demographic and socioeconomic groups.

Conclusions

This 2-phase study focuses on caregiver-based iterative development and evaluation of the quality and usability of the platform among caregivers. The development of our platform incorporates the needs and opinions of end users throughout the entire process to ensure the creation of an effective product. The platform is rigorously assessed regarding overall app quality and acceptability using validated tools in a pilot group and across diverse caregivers, respectively. The phase I results demonstrate that digital platforms, especially those offering personalized and comprehensive support, hold significant promise for supporting family caregivers of people living with dementia. By the end of the study, we expect to deliver a highly accessible digital platform designed to assist dementia caregivers in managing financial and legal challenges by linking them to personalized and contextually relevant information and resources in Texas. If the Olera.care platform demonstrates the usefulness and ease of use, we plan to collaborate with caregiving organizations to expand it nationally, addressing the needs of the growing population of dementia caregivers.

Acknowledgments

The authors express their gratitude for the support by the Center for Community Health and Aging at Texas A&M University and community partners. They are also thankful to all the study participants. This research was funded by the National Institute on Aging Small Business Innovation Research program (contract 1R44AG074116-01, Solicitation AG21-025). The findings and conclusions expressed in this study are those of the authors and do not necessarily reflect the views of the National Institute on Aging.

Data Availability

The data sets generated and analyzed during this study are not publicly available due to protection of study participants’ privacy, but de-identified data sets are available from the corresponding author on reasonable request.

Authors' Contributions

TF, LD, SL, and MGO contributed to the original proposal submitted to the National Institute on Aging. LD, QF, MNH, LF, and TF contributed to this manuscript’s first draft writing. SL and MGO contributed to supervision. All authors contributed to the critical revision of the manuscript.

Conflicts of Interest

TF and LD are executives and owners of Olera, Inc.

NIH peer review summary statement for original proposal.

Number of visitors who found Olera.care website from free traffic of search engines by June 1, 2024.

Number of pages indexed by Google by June 1, 2024.

Number of client requests by June 1, 2024.

• NA. 2024 Alzheimer's disease facts and figures. Alzheimers Dement. May 2024;20(5):3708-3821. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wolff JL, Spillman BC, Freedman VA, Kasper JD. A national profile of family and unpaid caregivers who assist older adults with health care activities. JAMA Intern Med. 2016;176(3):372-379. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Friedman EM, Shih RA, Langa KM, Hurd MD. US prevalence and predictors of informal caregiving for dementia. Health Aff (Millwood). Oct 2015;34(10):1637-1641. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kokorelias KM, Gignac MAM, Naglie G, Rittenberg N, MacKenzie J, D'Souza S, et al. A grounded theory study to identify caregiving phases and support needs across the Alzheimer's disease trajectory. Disabil Rehabil. Apr 2022;44(7):1050-1059. [ CrossRef ] [ Medline ]
• Alspaugh ME, Stephens MA, Townsend AL, Zarit SH, Greene R. Longitudinal patterns of risk for depression in dementia caregivers: objective and subjective primary stress as predictors. Psychol Aging. Mar 1999;14(1):34-43. [ CrossRef ] [ Medline ]
• Caregiving in the United States 2020. AARP and National Alliance for Caregiving. May 14, 2020. URL: https://doi.org/10.26419/ppi.00103.001 [accessed 2024-06-10]
• Nandi A, Counts N, Bröker J, Malik S, Chen S, Han R, et al. Cost of care for Alzheimer's disease and related dementias in the United States: 2016 to 2060. NPJ Aging. Feb 08, 2024;10(1):13. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fan Q, DuBose L, Ory MG, Lee S, Hoang M, Vennatt J, et al. Financial, legal, and functional challenges of providing care for people living with dementia and needs for a digital platform: interview study among family caregivers. JMIR Aging. Sep 05, 2023;6:e47577. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brodaty H, Donkin M. Family caregivers of people with dementia. Dialogues Clin Neurosci. 2009;11(2):217-228. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Caregiving in the United States 2015. AARP and National Alliance for Caregiving. Jun 4, 2015. URL: https://aarp.org/ppi/info-2015/caregiving-in-the-united-states-2015 [accessed 2024-06-10]
• George LK, Gwyther LP. Caregiver well-being: a multidimensional examination of family caregivers of demented adults. Gerontologist. Jun 1986;26(3):253-259. [ CrossRef ] [ Medline ]
• Gurland BJ, Wilder DE, Lantigua R, Stern Y, Chen J, Killeffer EH, et al. Rates of dementia in three ethnoracial groups. Int J Geriatr Psychiatry. Jun 1999;14(6):481-493. [ Medline ]
• Duran-Kiraç G, Uysal-Bozkir Ö, Uittenbroek R, van Hout H, Broese van Groenou MI. Accessibility of health care experienced by persons with dementia from ethnic minority groups and formal and informal caregivers: A scoping review of European literature. Dementia (London). Feb 2022;21(2):677-700. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Morgan E, Dyar C, Feinstein BA, Rose K. Sexual and gender minority differences in likelihood of being a caregiver and levels of caregiver strain in a sample of older adults. J Homosex. Aug 23, 2024;71(10):2287-2299. [ CrossRef ] [ Medline ]
• Beinart N, Weinman J, Wade D, Brady R. Caregiver burden and psychoeducational interventions in Alzheimer's disease: a review. Dement Geriatr Cogn Dis Extra. Jan 2012;2(1):638-648. [ FREE Full text ] [ CrossRef ] [ Medline ]
• National organizations, programs, and other resources for caregivers. American Association of Retired Persons (AARP). Oct 28, 2019. URL: https://www.aarp.org/caregiving/local/info-2019/national-resources-for-caregivers.html [accessed 2024-06-09]
• Bhargava Y, Baths V. Technology for dementia care: benefits, opportunities and concerns. J Global Health Rep. 2022;6:e2022056. [ CrossRef ]
• Wójcik D, Szczechowiak K, Konopka P, Owczarek M, Kuzia A, Rydlewska-Liszkowska I, et al. Informal dementia caregivers: current technology use and acceptance of technology in care. Int J Environ Res Public Health. Mar 19, 2021;18(6):3167. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Astell AJ, Bouranis N, Hoey J, Lindauer A, Mihailidis A, Nugent C, et al. Technology and dementia: the future is now. Dement Geriatr Cogn Disord. 2019;47(3):131-139. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kinchin I, Edwards L, Adrion E, Chen Y, Ashour A, Leroi I, et al. Care partner needs of people with neurodegenerative disorders: What are the needs, and how well do the current assessment tools capture these needs? A systematic meta-review. Int J Geriatr Psychiatry. May 24, 2022;37(7). [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guisado-Fernández E, Giunti G, Mackey LM, Blake C, Caulfield BM. Factors influencing the adoption of smart health technologies for people with dementia and their informal caregivers: scoping review and design framework. JMIR Aging. Apr 30, 2019;2(1):e12192. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gitlin LN, Marx K, Stanley IH, Hodgson N. Translating evidence-based dementia caregiving interventions into practice: state-of-the-science and next steps. Gerontologist. Apr 2015;55(2):210-226. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mittelman MS, Bartels SJ. Translating research into practice: case study of a community-based dementia caregiver intervention. Health Aff (Millwood). Apr 2014;33(4):587-595. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Foster MV, Sethares KA. Facilitators and barriers to the adoption of telehealth in older adults: an integrative review. Comput Inform Nurs. Nov 2014;32(11):523-535. [ CrossRef ] [ Medline ]
• Napoles AM, Chadiha L, Eversley R, Moreno-John G. Reviews: developing culturally sensitive dementia caregiver interventions: are we there yet? Am J Alzheimers Dis Other Demen. Aug 2010;25(5):389-406. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bastoni S, Wrede C, da Silva MC, Sanderman R, Gaggioli A, Braakman-Jansen A, et al. Factors influencing implementation of eHealth technologies to support informal dementia care: umbrella review. JMIR Aging. Oct 08, 2021;4(4):e30841. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;18(3):321-325. [ CrossRef ]
• Milella F, Russo DD, Bandini S. AI-powered solutions to support informal caregivers in their decision-making: a systematic review of the literature. OBM Geriatr. 2023;7(4):1-11. [ CrossRef ]
• Hird N, Osaki T, Ghosh S, Palaniappan SK, Maeda K. Enabling personalization for digital cognitive stimulation to support communication with people with dementia: pilot intervention study as a prelude to AI development. JMIR Form Res. Jan 16, 2024;8:e51732. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garcia-Ptacek S, Dahlrup B, Edlund AK, Wijk H, Eriksdotter M. The caregiving phenomenon and caregiver participation in dementia. Scand J Caring Sci. Jun 2019;33(2):255-265. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ambegaonkar A, Ritchie C, de la Fuente Garcia S. The use of mobile applications as communication aids for people with dementia: opportunities and limitations. J Alzheimers Dis Rep. 2021;5(1):681-692. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Martínez-Alcalá CI, Pliego-Pastrana P, Rosales-Lagarde A, Lopez-Noguerola JS, Molina-Trinidad EM. Information and communication technologies in the care of the elderly: systematic review of applications aimed at patients with dementia and caregivers. JMIR Rehabil Assist Technol. May 02, 2016;3(1):e6. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Burdine JN, McLeroy K, Blakely C, Wendel ML, Felix MRJ. Community-based participatory research and community health development. J Prim Prev. Apr 2010;31(1-2):1-7. [ CrossRef ] [ Medline ]
• Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19:173-202. [ CrossRef ] [ Medline ]
• Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. Mar 11, 2015;3(1):e27. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rahimi B, Nadri H, Lotfnezhad Afshar H, Timpka T. A systematic review of the Technology Acceptance Model in Health informatics. Appl Clin Inform. Jul 2018;9(3):604-634. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nadal C, Sas C, Doherty G. Technology acceptance in mobile health: scoping review of definitions, models, and measurement. J Med Internet Res. Jul 06, 2020;22(7):e17256. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319-340. [ CrossRef ]
• Mendez KJW, Budhathoki C, Labrique AB, Sadak T, Tanner EK, Han HR. Factors associated with intention to adopt mHealth apps among dementia caregivers with a chronic condition: cross-sectional, correlational study. JMIR Mhealth Uhealth. Aug 31, 2021;9(8):e27926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Blank S. The Four Steps to the Epiphany: Successful Strategies for Products that Win. New York, NY. John Wiley & Sons; 2020.
• Python Software Foundation. Python. 2020. URL: https://www.python.org/downloads/release/python-390/ [accessed 2024-06-09]
• Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. Sep 18, 2013;13:117. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mandracchia F, Llauradó E, Tarro L, Valls RM, Solà R. Mobile Phone Apps for Food Allergies or Intolerances in App Stores: Systematic Search and Quality Assessment Using the Mobile App Rating Scale (MARS). JMIR Mhealth Uhealth. Sep 16, 2020;8(9):e18339. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Alzheimer's Disease 8 Dementia Screening Interview (AD8). Alzheimer's Association. URL: https://alz.org/media/documents/ad8-dementia-screening.pdf [accessed 2024-06-09]
• Holden RJ, Karsh BT. The technology acceptance model: its past and its future in health care. J Biomed Inform. Feb 2010;43(1):159-172. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Cohen J. Statistical Power Analysis for the Behavioral Sciences. New York, NY. Academic Press; 2013.
• Gallagher-Thompson D, Solano N, Coon D, Areán P. Recruitment and retention of latino dementia family caregivers in intervention research: issues to face, lessons to learn. Gerontologist. Feb 2003;43(1):45-51. [ CrossRef ] [ Medline ]
• Shatenstein B, Kergoat M, Reid I. Issues in recruitment, retention, and data collection in a longitudinal nutrition study of community-dwelling older adults with early-stage Alzheimer's dementia. J Appl Gerontol. Mar 11, 2008;27(3):267-285. [ CrossRef ]
• Ritchie J, Lewis J, Micholls CM, Ormston R. Qualitative Research Practice: A Guide for Social Science Students and Researchers. Washington, DC. Sage Publications; 2013.
• Fan Q, Hoang MN, DuBose L, Ory MG, Vennatt J, Salha D, et al. The Olera.care digital caregiving assistance platform for dementia caregivers: preliminary evaluation study. JMIR Aging. Apr 17, 2024;7:e55132. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mishra RK, Park C, Momin AS, Rafaei NE, Kunik M, York MK, et al. Care4AD: a technology-driven platform for care coordination and management: acceptability study in dementia. Gerontology. 2023;69(2):227-238. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gilson A, Gassman M, Dodds D, Lombardo R, Ford Ii JH, Potteiger M. Refining a digital therapeutic platform for home care agencies in dementia care to elicit stakeholder feedback: focus group study with stakeholders. JMIR Aging. Mar 02, 2022;5(1):e32516. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bults M, van Leersum CM, Olthuis TJJ, Siebrand E, Malik Z, Liu L, et al. Acceptance of a digital assistant (Anne4Care) for older adult immigrants living with dementia: qualitative descriptive study. JMIR Aging. Apr 19, 2024;7:e50219. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boutilier JJ, Loganathar P, Linden A, Scheer E, Noejovich S, Elliott C, et al. A web-based platform (CareVirtue) to support caregivers of people living with Alzheimer disease and related dementias: mixed methods feasibility study. JMIR Aging. Aug 04, 2022;5(3):e36975. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gitlin LN, Bouranis N, Kern V, Koeuth S, Marx KA, McClure LA, et al. WeCareAdvisor, an online platform to help family caregivers manage dementia-related behavioral symptoms: an efficacy trial in the time of COVID-19. J Technol Behav Sci. 2022;7(1):33-44. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fuller-Tyszkiewicz M, Richardson B, Little K, Teague S, Hartley-Clark L, Capic T, et al. Efficacy of a smartphone app intervention for reducing caregiver stress: randomized controlled trial. JMIR Ment Health. Jul 24, 2020;7(7):e17541. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rodriguez K, Fugard M, Amini S, Smith G, Marasco D, Shatzer J, et al. Caregiver response to an online dementia and caregiver wellness education platform. J Alzheimers Dis Rep. 2021;5(1):433-442. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Oostra DL, Vos WL, Olde Rikkert MGM, Nieuwboer MS, Perry M. Digital resilience monitoring of informal caregivers of persons with dementia for early detection of overburden: Development and pilot testing. Int J Geriatr Psychiatry. Jan 2023;38(1):e5869. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Guisado-Fernandez E, Caulfield B, Silva PA, Mackey L, Singleton D, Leahy D, et al. Development of a caregivers' support platform (Connected Health Sustaining Home Stay in Dementia): protocol for a longitudinal observational mixed methods study. JMIR Res Protoc. Aug 28, 2019;8(8):13280. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zafeiridi P, Paulson K, Dunn R, Wolverson E, White C, Thorpe JA, et al. A web-based platform for people with memory problems and their caregivers (CAREGIVERSPRO-MMD): mixed-methods evaluation of usability. JMIR Form Res. Mar 12, 2018;2(1):e4. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Werner NE, Brown JC, Loganathar P, Holden RJ. Quality of mobile apps for care partners of people with Alzheimer disease and related dementias: mobile app rating scale evaluation. JMIR Mhealth Uhealth. Mar 29, 2022;10(3):e33863. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Assfaw AD, Reinschmidt KM, Teasdale TA, Stephens L, Kleszynski KL, Dwyer K. Assessing culturally tailored dementia interventions to support informal caregivers of people living with dementia (PLWD): a scoping review. J Racial Ethn Health Disparities. 2024. [ CrossRef ] [ Medline ]
• Lapid MI, Atherton PJ, Clark MM, Kung S, Sloan JA, Rummans TA. Cancer caregiver: perceived benefits of technology. Telemed J E Health. Nov 2015;21(11):893-902. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Efthymiou A, Middleton N, Charalambous A, Papastavrou E. The association of health literacy and electronic health literacy with self-efficacy, coping, and caregiving perceptions among carers of people with dementia: research protocol for a descriptive correlational study. JMIR Res Protoc. Nov 13, 2017;6(11):e221. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T Leung; The proposal for this study was peer reviewed by the Special Emphasis Panels of the Risk, Prevention and Health Behavior Integrated Review Group - Training and Education for Alzheimer's disease (AD) and AD-related dementias (ADRD) Caregivers on Financial Management and Legal Planning (National Institutes of Health, USA). See the Multimedia Appendix for the peer-review report; submitted 09.07.24; accepted 13.07.24; published 07.08.24.

©Logan DuBose, Qiping Fan, Louis Fisher, Minh-Nguyet Hoang, Diana Salha, Shinduk Lee, Marcia G Ory, Tokunbo Falohun. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 07.08.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.