evaluation of research papers

• Privacy Policy

Home » Evaluating Research – Process, Examples and Methods

Evaluating Research – Process, Examples and Methods

Table of Contents

Evaluating Research

Definition:

Evaluating Research refers to the process of assessing the quality, credibility, and relevance of a research study or project. This involves examining the methods, data, and results of the research in order to determine its validity, reliability, and usefulness. Evaluating research can be done by both experts and non-experts in the field, and involves critical thinking, analysis, and interpretation of the research findings.

Research Evaluating Process

The process of evaluating research typically involves the following steps:

Identify the Research Question

The first step in evaluating research is to identify the research question or problem that the study is addressing. This will help you to determine whether the study is relevant to your needs.

Assess the Study Design

The study design refers to the methodology used to conduct the research. You should assess whether the study design is appropriate for the research question and whether it is likely to produce reliable and valid results.

Evaluate the Sample

The sample refers to the group of participants or subjects who are included in the study. You should evaluate whether the sample size is adequate and whether the participants are representative of the population under study.

Review the Data Collection Methods

You should review the data collection methods used in the study to ensure that they are valid and reliable. This includes assessing the measures used to collect data and the procedures used to collect data.

Examine the Statistical Analysis

Statistical analysis refers to the methods used to analyze the data. You should examine whether the statistical analysis is appropriate for the research question and whether it is likely to produce valid and reliable results.

Assess the Conclusions

You should evaluate whether the data support the conclusions drawn from the study and whether they are relevant to the research question.

Consider the Limitations

Finally, you should consider the limitations of the study, including any potential biases or confounding factors that may have influenced the results.

Evaluating Research Methods

Evaluating Research Methods are as follows:

• Peer review: Peer review is a process where experts in the field review a study before it is published. This helps ensure that the study is accurate, valid, and relevant to the field.
• Critical appraisal : Critical appraisal involves systematically evaluating a study based on specific criteria. This helps assess the quality of the study and the reliability of the findings.
• Replication : Replication involves repeating a study to test the validity and reliability of the findings. This can help identify any errors or biases in the original study.
• Meta-analysis : Meta-analysis is a statistical method that combines the results of multiple studies to provide a more comprehensive understanding of a particular topic. This can help identify patterns or inconsistencies across studies.
• Consultation with experts : Consulting with experts in the field can provide valuable insights into the quality and relevance of a study. Experts can also help identify potential limitations or biases in the study.
• Review of funding sources: Examining the funding sources of a study can help identify any potential conflicts of interest or biases that may have influenced the study design or interpretation of results.

Example of Evaluating Research

Example of Evaluating Research sample for students:

Title of the Study: The Effects of Social Media Use on Mental Health among College Students

Sample Size: 500 college students

Sampling Technique : Convenience sampling

• Sample Size: The sample size of 500 college students is a moderate sample size, which could be considered representative of the college student population. However, it would be more representative if the sample size was larger, or if a random sampling technique was used.
• Sampling Technique : Convenience sampling is a non-probability sampling technique, which means that the sample may not be representative of the population. This technique may introduce bias into the study since the participants are self-selected and may not be representative of the entire college student population. Therefore, the results of this study may not be generalizable to other populations.
• Participant Characteristics: The study does not provide any information about the demographic characteristics of the participants, such as age, gender, race, or socioeconomic status. This information is important because social media use and mental health may vary among different demographic groups.
• Data Collection Method: The study used a self-administered survey to collect data. Self-administered surveys may be subject to response bias and may not accurately reflect participants’ actual behaviors and experiences.
• Data Analysis: The study used descriptive statistics and regression analysis to analyze the data. Descriptive statistics provide a summary of the data, while regression analysis is used to examine the relationship between two or more variables. However, the study did not provide information about the statistical significance of the results or the effect sizes.

Overall, while the study provides some insights into the relationship between social media use and mental health among college students, the use of a convenience sampling technique and the lack of information about participant characteristics limit the generalizability of the findings. In addition, the use of self-administered surveys may introduce bias into the study, and the lack of information about the statistical significance of the results limits the interpretation of the findings.

Note*: Above mentioned example is just a sample for students. Do not copy and paste directly into your assignment. Kindly do your own research for academic purposes.

Applications of Evaluating Research

Here are some of the applications of evaluating research:

• Identifying reliable sources : By evaluating research, researchers, students, and other professionals can identify the most reliable sources of information to use in their work. They can determine the quality of research studies, including the methodology, sample size, data analysis, and conclusions.
• Validating findings: Evaluating research can help to validate findings from previous studies. By examining the methodology and results of a study, researchers can determine if the findings are reliable and if they can be used to inform future research.
• Identifying knowledge gaps: Evaluating research can also help to identify gaps in current knowledge. By examining the existing literature on a topic, researchers can determine areas where more research is needed, and they can design studies to address these gaps.
• Improving research quality : Evaluating research can help to improve the quality of future research. By examining the strengths and weaknesses of previous studies, researchers can design better studies and avoid common pitfalls.
• Informing policy and decision-making : Evaluating research is crucial in informing policy and decision-making in many fields. By examining the evidence base for a particular issue, policymakers can make informed decisions that are supported by the best available evidence.
• Enhancing education : Evaluating research is essential in enhancing education. Educators can use research findings to improve teaching methods, curriculum development, and student outcomes.

Purpose of Evaluating Research

Here are some of the key purposes of evaluating research:

• Determine the reliability and validity of research findings : By evaluating research, researchers can determine the quality of the study design, data collection, and analysis. They can determine whether the findings are reliable, valid, and generalizable to other populations.
• Identify the strengths and weaknesses of research studies: Evaluating research helps to identify the strengths and weaknesses of research studies, including potential biases, confounding factors, and limitations. This information can help researchers to design better studies in the future.
• Inform evidence-based decision-making: Evaluating research is crucial in informing evidence-based decision-making in many fields, including healthcare, education, and public policy. Policymakers, educators, and clinicians rely on research evidence to make informed decisions.
• Identify research gaps : By evaluating research, researchers can identify gaps in the existing literature and design studies to address these gaps. This process can help to advance knowledge and improve the quality of research in a particular field.
• Ensure research ethics and integrity : Evaluating research helps to ensure that research studies are conducted ethically and with integrity. Researchers must adhere to ethical guidelines to protect the welfare and rights of study participants and to maintain the trust of the public.

Characteristics Evaluating Research

Characteristics Evaluating Research are as follows:

• Research question/hypothesis: A good research question or hypothesis should be clear, concise, and well-defined. It should address a significant problem or issue in the field and be grounded in relevant theory or prior research.
• Study design: The research design should be appropriate for answering the research question and be clearly described in the study. The study design should also minimize bias and confounding variables.
• Sampling : The sample should be representative of the population of interest and the sampling method should be appropriate for the research question and study design.
• Data collection : The data collection methods should be reliable and valid, and the data should be accurately recorded and analyzed.
• Results : The results should be presented clearly and accurately, and the statistical analysis should be appropriate for the research question and study design.
• Interpretation of results : The interpretation of the results should be based on the data and not influenced by personal biases or preconceptions.
• Generalizability: The study findings should be generalizable to the population of interest and relevant to other settings or contexts.
• Contribution to the field : The study should make a significant contribution to the field and advance our understanding of the research question or issue.

Advantages of Evaluating Research

Evaluating research has several advantages, including:

• Ensuring accuracy and validity : By evaluating research, we can ensure that the research is accurate, valid, and reliable. This ensures that the findings are trustworthy and can be used to inform decision-making.
• Identifying gaps in knowledge : Evaluating research can help identify gaps in knowledge and areas where further research is needed. This can guide future research and help build a stronger evidence base.
• Promoting critical thinking: Evaluating research requires critical thinking skills, which can be applied in other areas of life. By evaluating research, individuals can develop their critical thinking skills and become more discerning consumers of information.
• Improving the quality of research : Evaluating research can help improve the quality of research by identifying areas where improvements can be made. This can lead to more rigorous research methods and better-quality research.
• Informing decision-making: By evaluating research, we can make informed decisions based on the evidence. This is particularly important in fields such as medicine and public health, where decisions can have significant consequences.
• Advancing the field : Evaluating research can help advance the field by identifying new research questions and areas of inquiry. This can lead to the development of new theories and the refinement of existing ones.

Limitations of Evaluating Research

Limitations of Evaluating Research are as follows:

• Time-consuming: Evaluating research can be time-consuming, particularly if the study is complex or requires specialized knowledge. This can be a barrier for individuals who are not experts in the field or who have limited time.
• Subjectivity : Evaluating research can be subjective, as different individuals may have different interpretations of the same study. This can lead to inconsistencies in the evaluation process and make it difficult to compare studies.
• Limited generalizability: The findings of a study may not be generalizable to other populations or contexts. This limits the usefulness of the study and may make it difficult to apply the findings to other settings.
• Publication bias: Research that does not find significant results may be less likely to be published, which can create a bias in the published literature. This can limit the amount of information available for evaluation.
• Lack of transparency: Some studies may not provide enough detail about their methods or results, making it difficult to evaluate their quality or validity.
• Funding bias : Research funded by particular organizations or industries may be biased towards the interests of the funder. This can influence the study design, methods, and interpretation of results.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Data Verification – Process, Types and Examples

Tables in Research Paper – Types, Creating Guide...

Literature Review – Types Writing Guide and...

Data Interpretation – Process, Methods and...

Future Research – Thesis Guide

Appendices – Writing Guide, Types and Examples

Research Evaluation

• First Online: 23 June 2020

Cite this chapter

• Carlo Ghezzi 2  

1050 Accesses

1 Citations

• The original version of this chapter was revised. A correction to this chapter can be found at https://doi.org/10.1007/978-3-030-45157-8_7

This chapter is about research evaluation. Evaluation is quintessential to research. It is traditionally performed through qualitative expert judgement. The chapter presents the main evaluation activities in which researchers can be engaged. It also introduces the current efforts towards devising quantitative research evaluation based on bibliometric indicators and critically discusses their limitations, along with their possible (limited and careful) use.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as EPUB and PDF
• Read on any device
• Instant download
• Own it forever
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Change history

19 october 2021.

The original version of the chapter was inadvertently published with an error. The chapter has now been corrected.

Notice that the taxonomy presented in Box 5.1 does not cover all kinds of scientific papers. As an example, it does not cover survey papers, which normally are not submitted to a conference.

Private institutions and industry may follow different schemes.

Adler, R., Ewing, J., Taylor, P.: Citation statistics: A report from the international mathematical union (imu) in cooperation with the international council of industrial and applied mathematics (iciam) and the institute of mathematical statistics (ims). Statistical Science 24 (1), 1–14 (2009). URL http://www.jstor.org/stable/20697661

Esposito, F., Ghezzi, C., Hermenegildo, M., Kirchner, H., Ong, L.: Informatics Research Evaluation. Informatics Europe (2018). URL https://www.informatics-europe.org/publications.html

Friedman, B., Schneider, F.B.: Incentivizing quality and impact: Evaluating scholarship in hiring, tenure, and promotion. Computing Research Association (2016). URL https://cra.org/resources/best-practice-memos/incentivizing-quality-and-impact-evaluating-scholarship-in-hiring-tenure-and-promotion/

Hicks, D., Wouters, P., Waltman, L., de Rijcke, S., Rafols, I.: Bibliometrics: The leiden manifesto for research metrics. Nature News 520 (7548), 429 (2015). https://doi.org/10.1038/520429a . URL http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351

Parnas, D.L.: Stop the numbers game. Commun. ACM 50 (11), 19–21 (2007). https://doi.org/10.1145/1297797.1297815 . URL http://doi.acm.org/10.1145/1297797.1297815

Patterson, D., Snyder, L., Ullman, J.: Evaluating computer scientists and engineers for promotion and tenure. Computing Research Association (1999). URL https://cra.org/resources/best-practice-memos/incentivizing-quality-and-impact-evaluating-scholarship-in-hiring-tenure-and-promotion/

Saenen, B., Borrell-Damian, L.: Reflections on University Research Assessment: key concepts, issues and actors. European University Association (2019). URL https://eua.eu/component/attachments/attachments.html?id=2144

Download references

Author information

Authors and affiliations.

Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milano, Italy

Carlo Ghezzi

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Carlo Ghezzi .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Ghezzi, C. (2020). Research Evaluation. In: Being a Researcher. Springer, Cham. https://doi.org/10.1007/978-3-030-45157-8_5

Download citation

DOI : https://doi.org/10.1007/978-3-030-45157-8_5

Published : 23 June 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-45156-1

Online ISBN : 978-3-030-45157-8

eBook Packages : Computer Science Computer Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Search Menu
• Sign in through your institution
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

1. introduction, what is meant by impact, 2. why evaluate research impact, 3. evaluating research impact, 4. impact and the ref, 5. the challenges of impact evaluation, 6. developing systems and taxonomies for capturing impact, 7. indicators, evidence, and impact within systems, 8. conclusions and recommendations.

• < Previous

Assessment, evaluations, and definitions of research impact: A review

• Article contents
• Figures & tables
• Supplementary Data

Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation , Volume 23, Issue 1, January 2014, Pages 21–32, https://doi.org/10.1093/reseval/rvt021

• Permissions Icon Permissions

This article aims to explore what is understood by the term ‘research impact’ and to provide a comprehensive assimilation of available literature and information, drawing on global experiences to understand the potential for methods and frameworks of impact assessment being implemented for UK impact assessment. We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems.

When considering the impact that is generated as a result of research, a number of authors and government recommendations have advised that a clear definition of impact is required ( Duryea, Hochman, and Parfitt 2007 ; Grant et al. 2009 ; Russell Group 2009 ). From the outset, we note that the understanding of the term impact differs between users and audiences. There is a distinction between ‘academic impact’ understood as the intellectual contribution to one’s field of study within academia and ‘external socio-economic impact’ beyond academia. In the UK, evaluation of academic and broader socio-economic impact takes place separately. ‘Impact’ has become the term of choice in the UK for research influence beyond academia. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research.

an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia

Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. As such research outputs, for example, knowledge generated and publications, can be translated into outcomes, for example, new products and services, and impacts or added value ( Duryea et al. 2007 ). Although some might find the distinction somewhat marginal or even confusing, this differentiation between outputs, outcomes, and impacts is important, and has been highlighted, not only for the impacts derived from university research ( Kelly and McNicol 2011 ) but also for work done in the charitable sector ( Ebrahim and Rangan, 2010 ; Berg and Månsson 2011 ; Kelly and McNicoll 2011 ). The Social Return on Investment (SROI) guide ( The SROI Network 2012 ) suggests that ‘The language varies “impact”, “returns”, “benefits”, “value” but the questions around what sort of difference and how much of a difference we are making are the same’. It is perhaps assumed here that a positive or beneficial effect will be considered as an impact but what about changes that are perceived to be negative? Wooding et al. (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from ‘benefit’ to ‘impact’ when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. Thalidomide has since been found to have beneficial effects in the treatment of certain types of cancer. Clearly the impact of thalidomide would have been viewed very differently in the 1950s compared with the 1960s or today.

In viewing impact evaluations it is important to consider not only who has evaluated the work but the purpose of the evaluation to determine the limits and relevance of an assessment exercise. In this article, we draw on a broad range of examples with a focus on methods of evaluation for research impact within Higher Education Institutions (HEIs). As part of this review, we aim to explore the following questions:

What are the reasons behind trying to understand and evaluate research impact?

What are the methodologies and frameworks that have been employed globally to assess research impact and how do these compare?

What are the challenges associated with understanding and evaluating research impact?

What indicators, evidence, and impacts need to be captured within developing systems

What are the reasons behind trying to understand and evaluate research impact? Throughout history, the activities of a university have been to provide both education and research, but the fundamental purpose of a university was perhaps described in the writings of mathematician and philosopher Alfred North Whitehead (1929) .

‘The justification for a university is that it preserves the connection between knowledge and the zest of life, by uniting the young and the old in the imaginative consideration of learning. The university imparts information, but it imparts it imaginatively. At least, this is the function which it should perform for society. A university which fails in this respect has no reason for existence. This atmosphere of excitement, arising from imaginative consideration transforms knowledge.’

In undertaking excellent research, we anticipate that great things will come and as such one of the fundamental reasons for undertaking research is that we will generate and transform knowledge that will benefit society as a whole.

One might consider that by funding excellent research, impacts (including those that are unforeseen) will follow, and traditionally, assessment of university research focused on academic quality and productivity. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support ( Kelly and McNicoll 2011 ).

The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes.

HEIs overview. To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities.

Accountability. To demonstrate to government, stakeholders, and the wider public the value of research. There has been a drive from the UK government through Higher Education Funding Council for England (HEFCE) and the Research Councils ( HM Treasury 2004 ) to account for the spending of public money by demonstrating the value of research to tax payers, voters, and the public in terms of socio-economic benefits ( European Science Foundation 2009 ), in effect, justifying this expenditure ( Davies Nutley, and Walter 2005 ; Hanney and González-Block 2011 ).

Inform funding. To understand the socio-economic value of research and subsequently inform funding decisions. By evaluating the contribution that research makes to society and the economy, future funding can be allocated where it is perceived to bring about the desired impact. As Donovan (2011) comments, ‘Impact is a strong weapon for making an evidence based case to governments for enhanced research support’.

Understand. To understand the method and routes by which research leads to impacts to maximize on the findings that come out of research and develop better ways of delivering impact.

The growing trend for accountability within the university system is not limited to research and is mirrored in assessments of teaching quality, which now feed into evaluation of universities to ensure fee-paying students’ satisfaction. In demonstrating research impact, we can provide accountability upwards to funders and downwards to users on a project and strategic basis ( Kelly and McNicoll 2011 ). Organizations may be interested in reviewing and assessing research impact for one or more of the aforementioned purposes and this will influence the way in which evaluation is approached.

It is important to emphasize that ‘Not everyone within the higher education sector itself is convinced that evaluation of higher education activity is a worthwhile task’ ( Kelly and McNicoll 2011 ). The University and College Union ( University and College Union 2011 ) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society ( University and College Union 2011 ). Impact assessments raise concerns over the steer of research towards disciplines and topics in which impact is more easily evidenced and that provide economic impacts that could subsequently lead to a devaluation of ‘blue skies’ research. Johnston ( Johnston 1995 ) notes that by developing relationships between researchers and industry, new research strategies can be developed. This raises the questions of whether UK business and industry should not invest in the research that will deliver them impacts and who will fund basic research if not the government? Donovan (2011) asserts that there should be no disincentive for conducting basic research. By asking academics to consider the impact of the research they undertake and by reviewing and funding them accordingly, the result may be to compromise research by steering it away from the imaginative and creative quest for knowledge. Professor James Ladyman, at the University of Bristol, a vocal adversary of awarding funding based on the assessment of research impact, has been quoted as saying that ‘…inclusion of impact in the REF will create “selection pressure,” promoting academic research that has “more direct economic impact” or which is easier to explain to the public’ ( Corbyn 2009 ).

Despite the concerns raised, the broader socio-economic impacts of research will be included and count for 20% of the overall research assessment, as part of the REF in 2014. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment.

What are the methodologies and frameworks that have been employed globally to evaluate research impact and how do these compare? The traditional form of evaluation of university research in the UK was based on measuring academic impact and quality through a process of peer review ( Grant 2006 ). Evidence of academic impact may be derived through various bibliometric methods, one example of which is the H index, which has incorporated factors such as the number of publications and citations. These metrics may be used in the UK to understand the benefits of research within academia and are often incorporated into the broader perspective of impact seen internationally, for example, within the Excellence in Research for Australia and using Star Metrics in the USA, in which quantitative measures are used to assess impact, for example, publications, citation, and research income. These ‘traditional’ bibliometric techniques can be regarded as giving only a partial picture of full impact ( Bornmann and Marx 2013 ) with no link to causality. Standard approaches actively used in programme evaluation such as surveys, case studies, bibliometrics, econometrics and statistical analyses, content analysis, and expert judgment are each considered by some (Vonortas and Link, 2012) to have shortcomings when used to measure impacts.

Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated ( Australian Research Council 2008 ). In the UK, more sophisticated assessments of impact incorporating wider socio-economic benefits were first investigated within the fields of Biomedical and Health Sciences ( Grant 2006 ), an area of research that wanted to be able to justify the significant investment it received. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. As a result, numerous and widely varying models and frameworks for assessing impact exist. Here we outline a few of the most notable models that demonstrate the contrast in approaches available.

The Payback Framework is possibly the most widely used and adapted model for impact assessment ( Wooding et al. 2007 ; Nason et al. 2008 ), developed during the mid-1990s by Buxton and Hanney, working at Brunel University. It incorporates both academic outputs and wider societal benefits ( Donovan and Hanney 2011 ) to assess outcomes of health sciences research. The Payback Framework systematically links research with the associated benefits ( Scoble et al. 2010 ; Hanney and González-Block 2011 ) and can be thought of in two parts: a model that allows the research and subsequent dissemination process to be broken into specific components within which the benefits of research can be studied, and second, a multi-dimensional classification scheme into which the various outputs, outcomes, and impacts can be placed ( Hanney and Gonzalez Block 2011 ). The Payback Framework has been adopted internationally, largely within the health sector, by organizations such as the Canadian Institute of Health Research, the Dutch Public Health Authority, the Australian National Health and Medical Research Council, and the Welfare Bureau in Hong Kong ( Bernstein et al. 2006 ; Nason et al. 2008 ; CAHS 2009; Spaapen et al. n.d. ). The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. For more extensive reviews of the Payback Framework, see Davies et al. (2005) , Wooding et al. (2007) , Nason et al. (2008) , and Hanney and González-Block (2011) .

A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing ‘productive interactions’ between researchers and stakeholders by analysing the networks that evolve during research programmes ( Spaapen and Drooge, 2011 ; Spaapen et al. n.d. ). SIAMPI is based on the widely held assumption that interactions between researchers and stakeholder are an important pre-requisite to achieving impact ( Donovan 2011 ; Hughes and Martin 2012 ; Spaapen et al. n.d. ). This framework is intended to be used as a learning tool to develop a better understanding of how research interactions lead to social impact rather than as an assessment tool for judging, showcasing, or even linking impact to a specific piece of research. SIAMPI has been used within the Netherlands Institute for health Services Research ( SIAMPI n.d. ). ‘Productive interactions’, which can perhaps be viewed as instances of knowledge exchange, are widely valued and supported internationally as mechanisms for enabling impact and are often supported financially for example by Canada’s Social Sciences and Humanities Research Council, which aims to support knowledge exchange (financially) with a view to enabling long-term impact. In the UK, UK Department for Business, Innovation, and Skills provided funding of £150 million for knowledge exchange in 2011–12 to ‘help universities and colleges support the economic recovery and growth, and contribute to wider society’ ( Department for Business, Innovation and Skills 2012 ). While valuing and supporting knowledge exchange is important, SIAMPI perhaps takes this a step further in enabling these exchange events to be captured and analysed. One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. A comprehensive assessment of impact itself is not undertaken with SIAMPI, which make it a less-suitable method where showcasing the benefits of research is desirable or where this justification of funding based on impact is required.

The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. The RQF was developed to demonstrate and justify public expenditure on research, and as part of this framework, a pilot assessment was undertaken by the Australian Technology Network. Researchers were asked to evidence the economic, societal, environmental, and cultural impact of their research within broad categories, which were then verified by an expert panel ( Duryea et al. 2007 ) who concluded that the researchers and case studies could provide enough qualitative and quantitative evidence for reviewers to assess the impact arising from their research ( Duryea et al. 2007 ). To evaluate impact, case studies were interrogated and verifiable indicators assessed to determine whether research had led to reciprocal engagement, adoption of research findings, or public value. The RQF pioneered the case study approach to assessing research impact; however, with a change in government in 2007, this framework was never implemented in Australia, although it has since been taken up and adapted for the UK REF.

In developing the UK REF, HEFCE commissioned a report, in 2009, from RAND to review international practice for assessing research impact and provide recommendations to inform the development of the REF. RAND selected four frameworks to represent the international arena ( Grant et al. 2009 ). One of these, the RQF, they identified as providing a ‘promising basis for developing an impact approach for the REF’ using the case study approach. HEFCE developed an initial methodology that was then tested through a pilot exercise. The case study approach, recommended by the RQF, was combined with ‘significance’ and ‘reach’ as criteria for assessment. The criteria for assessment were also supported by a model developed by Brunel for ‘measurement’ of impact that used similar measures defined as depth and spread. In the Brunel model, depth refers to the degree to which the research has influenced or caused change, whereas spread refers to the extent to which the change has occurred and influenced end users. Evaluation of impact in terms of reach and significance allows all disciplines of research and types of impact to be assessed side-by-side ( Scoble et al. 2010 ).

The range and diversity of frameworks developed reflect the variation in purpose of evaluation including the stakeholders for whom the assessment takes place, along with the type of impact and evidence anticipated. The most appropriate type of evaluation will vary according to the stakeholder whom we are wishing to inform. Studies ( Buxton, Hanney and Jones 2004 ) into the economic gains from biomedical and health sciences determined that different methodologies provide different ways of considering economic benefits. A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006) .

Evaluation of impact is becoming increasingly important, both within the UK and internationally, and research and development into impact evaluation continues, for example, researchers at Brunel have developed the concept of depth and spread further into the Brunel Impact Device for Evaluation, which also assesses the degree of separation between research and impact ( Scoble et al. working paper ).

Although based on the RQF, the REF did not adopt all of the suggestions held within, for example, the option of allowing research groups to opt out of impact assessment should the nature or stage of research deem it unsuitable ( Donovan 2008 ). In 2009–10, the REF team conducted a pilot study for the REF involving 29 institutions, submitting case studies to one of five units of assessment (in clinical medicine, physics, earth systems and environmental sciences, social work and social policy, and English language and literature) ( REF2014 2010 ). These case studies were reviewed by expert panels and, as with the RQF, they found that it was possible to assess impact and develop ‘impact profiles’ using the case study approach ( REF2014 2010 ).

From 2014, research within UK universities and institutions will be assessed through the REF; this will replace the Research Assessment Exercise, which has been used to assess UK research since the 1980s. Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. The REF will therefore assess three aspects of research:

Environment

Research impact is assessed in two formats, first, through an impact template that describes the approach to enabling impact within a unit of assessment, and second, using impact case studies that describe the impact taking place following excellent research within a unit of assessment ( REF2014 2011a ). HEFCE indicated that impact should merit a 25% weighting within the REF ( REF2014 2011b ); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% ( Russell Group 2009 ; Jump 2011 ) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments ( REF2014 2010 ).

The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. In the UK, evidence and research impacts will be assessed for the REF within research disciplines. Although it can be envisaged that the range of impacts derived from research of different disciplines are likely to vary, one might question whether it makes sense to compare impacts within disciplines when the range of impact can vary enormously, for example, from business development to cultural changes or saving lives? An alternative approach was suggested for the RQF in Australia, where it was proposed that types of impact be compared rather than impact from specific disciplines.

Providing advice and guidance within specific disciplines is undoubtedly helpful. It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline ( REF2014 2012 ). Why should this be the case? Two areas of research impact health and biomedical sciences and the social sciences have received particular attention in the literature by comparison with, for example, the arts. Reviews and guidance on developing and evidencing impact in particular disciplines include the London School of Economics (LSE) Public Policy Group’s impact handbook (LSE n.d.), a review of the social and economic impacts arising from the arts produced by Reeve ( Reeves 2002 ), and a review by Kuruvilla et al. (2006) on the impact arising from health research. Perhaps it is time for a generic guide based on types of impact rather than research discipline?

What are the challenges associated with understanding and evaluating research impact? In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. Given that the type of impact we might expect varies according to research discipline, impact-specific challenges present us with the problem that an evaluation mechanism may not fairly compare impact between research disciplines.

5.1 Time lag

The time lag between research and impact varies enormously. For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. In development of the RQF, The Allen Consulting Group (2005) highlighted that defining a time lag between research and impact was difficult. In the UK, the Russell Group Universities responded to the REF consultation by recommending that no time lag be put on the delivery of impact from a piece of research citing examples such as the development of cardiovascular disease treatments, which take between 10 and 25 years from research to impact ( Russell Group 2009 ). To be considered for inclusion within the REF, impact must be underpinned by research that took place between 1 January 1993 and 31 December 2013, with impact occurring during an assessment window from 1 January 2008 to 31 July 2013. However, there has been recognition that this time window may be insufficient in some instances, with architecture being granted an additional 5-year period ( REF2014 2012 ); why only architecture has been granted this dispensation is not clear, when similar cases could be made for medicine, physics, or even English literature. Recommendations from the REF pilot were that the panel should be able to extend the time frame where appropriate; this, however, poses difficult decisions when submitting a case study to the REF as to what the view of the panel will be and whether if deemed inappropriate this will render the case study ‘unclassified’.

5.2 The developmental nature of impact

Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. Impact can be temporary or long-lasting. The point at which assessment takes place will therefore influence the degree and significance of that impact. For example, following the discovery of a new potential drug, preclinical work is required, followed by Phase 1, 2, and 3 trials, and then regulatory approval is granted before the drug is used to deliver potential health benefits. Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. If impact is short-lived and has come and gone within an assessment period, how will it be viewed and considered? Again the objective and perspective of the individuals and organizations assessing impact will be key to understanding how temporal and dissipated impact will be valued in comparison with longer-term impact.

5.3 Attribution

Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. The exploitation of research to provide impact occurs through a complex variety of processes, individuals, and organizations, and therefore, attributing the contribution made by a specific individual, piece of research, funding, strategy, or organization to an impact is not straight forward. Husbands-Fealing suggests that to assist identification of causality for impact assessment, it is useful to develop a theoretical framework to map the actors, activities, linkages, outputs, and impacts within the system under evaluation, which shows how later phases result from earlier ones. Such a framework should be not linear but recursive, including elements from contextual environments that influence and/or interact with various aspects of the system. Impact is often the culmination of work within spanning research communities ( Duryea et al. 2007 ). Concerns over how to attribute impacts have been raised many times ( The Allen Consulting Group 2005 ; Duryea et al. 2007 ; Grant et al. 2009 ), and differentiating between the various major and minor contributions that lead to impact is a significant challenge.

Figure 1 , replicated from Hughes and Martin (2012) , illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research.

Time, attribution, impact. Replicated from ( Hughes and Martin 2012 ).

This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. Research findings will be taken up in other branches of research and developed further before socio-economic impact occurs, by which point, attribution becomes a huge challenge. If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that ‘one is able to see further by standing on the shoulders of giants’.

5.4 Knowledge creep

It is acknowledged that one of the outcomes of developing new knowledge through research can be ‘knowledge creep’ where new data or information becomes accepted and gets absorbed over time. This is particularly recognized in the development of new government policy where findings can influence policy debate and policy change, without recognition of the contributing research ( Davies et al. 2005 ; Wooding et al. 2007 ). This is recognized as being particularly problematic within the social sciences where informing policy is a likely impact of research. In putting together evidence for the REF, impact can be attributed to a specific piece of research if it made a ‘distinctive contribution’ ( REF2014 2011a ). The difficulty then is how to determine what the contribution has been in the absence of adequate evidence and how we ensure that research that results in impacts that cannot be evidenced is valued and supported.

5.5 Gathering evidence

Gathering evidence of the links between research and impact is not only a challenge where that evidence is lacking. The introduction of impact assessments with the requirement to collate evidence retrospectively poses difficulties because evidence, measurements, and baselines have, in many cases, not been collected and may no longer be available. While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle.

Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. Decker et al. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. It is desirable that the assignation of administrative tasks to researchers is limited, and therefore, to assist the tracking and collating of impact data, systems are being developed involving numerous projects and developments internationally, including Star Metrics in the USA, the ERC (European Research Council) Research Information System, and Lattes in Brazil ( Lane 2010 ; Mugabushaka and Papazoglou 2012 ).

Ideally, systems within universities internationally would be able to share data allowing direct comparisons, accurate storage of information developed in collaborations, and transfer of comparable data as researchers move between institutions. To achieve compatible systems, a shared language is required. CERIF (Common European Research Information Format) was developed for this purpose, first released in 1991; a number of projects and systems across Europe such as the ERC Research Information System ( Mugabushaka and Papazoglou 2012 ) are being developed as CERIF-compatible.

In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. However, the Achilles heel of any such attempt, as critics suggest, is the creation of a system that rewards what it can measure and codify, with the knock-on effect of directing research projects to deliver within the measures and categories that reward.

Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at King’s College London or outlined in relevant publications (MICE Project n.d.). A taxonomy of impact categories was then produced onto which impact could be mapped. What emerged on testing the MICE taxonomy ( Cooke and Nadim 2011 ), by mapping impacts from case studies, was that detailed categorization of impact was found to be too prescriptive. Every piece of research results in a unique tapestry of impact and despite the MICE taxonomy having more than 100 indicators, it was found that these did not suffice. It is perhaps worth noting that the expert panels, who assessed the pilot exercise for the REF, commented that the evidence provided by research institutes to demonstrate impact were ‘a unique collection’. Where quantitative data were available, for example, audience numbers or book sales, these numbers rarely reflected the degree of impact, as no context or baseline was available. Cooke and Nadim (2011) also noted that using a linear-style taxonomy did not reflect the complex networks of impacts that are generally found. The Goldsmith report ( Cooke and Nadim 2011 ) recommended making indicators ‘value free’, enabling the value or quality to be established in an impact descriptor that could be assessed by expert panels. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention.

While defining the terminology used to understand impact and indicators will enable comparable data to be stored and shared between organizations, we would recommend that any categorization of impacts be flexible such that impacts arising from non-standard routes can be placed. It is worth considering the degree to which indicators are defined and provide broader definitions with greater flexibility.

It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary ( Davies et al. 2005 ). Any tool for impact evaluation needs to be flexible, such that it enables access to impact data for a variety of purposes (Scoble et al. n.d.). Systems need to be able to capture links between and evidence of the full pathway from research to impact, including knowledge exchange, outputs, outcomes, and interim impacts, to allow the route to impact to be traced. This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway.

Baselines and controls need to be captured alongside change to demonstrate the degree of impact. In many instances, controls are not feasible as we cannot look at what impact would have occurred if a piece of research had not taken place; however, indications of the picture before and after impact are valuable and worth collecting for impact that can be predicted.

It is now possible to use data-mining tools to extract specific data from narratives or unstructured data ( Mugabushaka and Papazoglou 2012 ). This is being done for collation of academic impact and outputs, for example, Research Portfolio Online Reporting Tools, which uses PubMed and text mining to cluster research projects, and STAR Metrics in the US, which uses administrative records and research outputs and is also being implemented by the ERC using data in the public domain ( Mugabushaka and Papazoglou 2012 ). These techniques have the potential to provide a transformation in data capture and impact assessment ( Jones and Grant 2013 ). It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. For systems to be able to capture a full range of systems, definitions and categories of impact need to be determined that can be incorporated into system development. To adequately capture interactions taking place between researchers, institutions, and stakeholders, the introduction of tools to enable this would be very valuable. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. Capturing knowledge exchange events would greatly assist the linking of research with impact.

The transition to routine capture of impact data not only requires the development of tools and systems to help with implementation but also a cultural change to develop practices, currently undertaken by a few to be incorporated as standard behaviour among researchers and universities.

What indicators, evidence, and impacts need to be captured within developing systems? There is a great deal of interest in collating terms for impact and indicators of impact. Consortia for Advancing Standards in Research Administration Information, for example, has put together a data dictionary with the aim of setting the standards for terminology used to describe impact and indicators that can be incorporated into systems internationally and seems to be building a certain momentum in this area. A variety of types of indicators can be captured within systems; however, it is important that these are universally understood. Here we address types of evidence that need to be captured to enable an overview of impact to be developed. In the majority of cases, a number of types of evidence will be required to provide an overview of impact.

7.1 Metrics

Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on. Metrics in themselves cannot convey the full impact; however, they are often viewed as powerful and unequivocal forms of evidence. If metrics are available as impact evidence, they should, where possible, also capture any baseline or control data. Any information on the context of the data will be valuable to understanding the degree to which impact has taken place.

Perhaps, SROI indicates the desire to be able to demonstrate the monetary value of investment and impact by some organizations. SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. This is a metric that has been used within the charitable sector ( Berg and Månsson 2011 ) and also features as evidence in the REF guidance for panel D ( REF2014 2012 ). More details on SROI can be found in ‘A guide to Social Return on Investment’ produced by The SROI Network (2012) .

Although metrics can provide evidence of quantitative changes or impacts from our research, they are unable to adequately provide evidence of the qualitative impacts that take place and hence are not suitable for all of the impact we will encounter. The main risks associated with the use of standardized metrics are that

The full impact will not be realized, as we focus on easily quantifiable indicators

We will focus attention towards generating results that enable boxes to be ticked rather than delivering real value for money and innovative research.

They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital.

7.2 Narratives

Narratives can be used to describe impact; the use of narratives enables a story to be told and the impact to be placed in context and can make good use of qualitative information. They are often written with a reader from a particular stakeholder group in mind and will present a view of impact from a particular perspective. The risk of relying on narratives to assess impact is that they often lack the evidence required to judge whether the research and impact are linked appropriately. Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. Table 1 summarizes some of the advantages and disadvantages of the case study approach.

The advantages and disadvantages of the case study approach

By allowing impact to be placed in context, we answer the ‘so what?’ question that can result from quantitative data analyses, but is there a risk that the full picture may not be presented to demonstrate impact in a positive light? Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact?

7.3 Surveys and testimonies

One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed.

The ability to record and log these type of data is important for enabling the path from research to impact to be established and the development of systems that can capture this would be very valuable.

7.4 Citations (outside of academia) and documentation

Citations (outside of academia) and documentation can be used as evidence to demonstrate the use research findings in developing new ideas and products for example. This might include the citation of a piece of research in policy documents or reference to a piece of research being cited within the media. A collation of several indicators of impact may be enough to convince that an impact has taken place. Even where we can evidence changes and benefits linked to our research, understanding the causal relationship may be difficult. Media coverage is a useful means of disseminating our research and ideas and may be considered alongside other evidence as contributing to or an indicator of impact.

The fast-moving developments in the field of altmetrics (or alternative metrics) are providing a richer understanding of how research is being used, viewed, and moved. The transfer of information electronically can be traced and reviewed to provide data on where and to whom research findings are going.

The understanding of the term impact varies considerably and as such the objectives of an impact assessment need to be thoroughly understood before evidence is collated.

While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context. While the case study is a useful way of showcasing impact, its limitations must be understood if we are to use this for evaluation purposes. The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. It is time-intensive to both assimilate and review case studies and we therefore need to ensure that the resources required for this type of evaluation are justified by the knowledge gained. The ability to write a persuasive well-evidenced case study may influence the assessment of impact. Over the past year, there have been a number of new posts created within universities, such as writing impact case studies, and a number of companies are now offering this as a contract service. A key concern here is that we could find that universities which can afford to employ either consultants or impact ‘administrators’ will generate the best case studies.

The development of tools and systems for assisting with impact evaluation would be very valuable. We suggest that developing systems that focus on recording impact information alone will not provide all that is required to link research to ensuing events and impacts, systems require the capacity to capture any interactions between researchers, the institution, and external stakeholders and link these with research findings and outputs or interim impacts to provide a network of data. In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. However, it must be remembered that in the case of the UK REF, impact is only considered that is based on research that has taken place within the institution submitting the case study. It is therefore in an institution’s interest to have a process by which all the necessary information is captured to enable a story to be developed in the absence of a researcher who may have left the employment of the institution. Figure 2 demonstrates the information that systems will need to capture and link.

Research findings including outputs (e.g., presentations and publications)

Communications and interactions with stakeholders and the wider public (emails, visits, workshops, media publicity, etc)

Feedback from stakeholders and communication summaries (e.g., testimonials and altmetrics)

Research developments (based on stakeholder input and discussions)

Outcomes (e.g., commercial and cultural, citations)

Impacts (changes, e.g., behavioural and economic)

Overview of the types of information that systems need to capture and link.

Attempting to evaluate impact to justify expenditure, showcase our work, and inform future funding decisions will only prove to be a valuable use of time and resources if we can take measures to ensure that assessment attempts will not ultimately have a negative influence on the impact of our research. There are areas of basic research where the impacts are so far removed from the research or are impractical to demonstrate; in these cases, it might be prudent to accept the limitations of impact assessment, and provide the potential for exclusion in appropriate circumstances.

This work was supported by Jisc [DIINN10].

Google Scholar

Google Preview

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Research Paper: A step-by-step guide: 7. Evaluating Sources

• 1. Getting Started
• 2. Topic Ideas
• 3. Thesis Statement & Outline
• 4. Appropriate Sources
• 5. Search Techniques
• 6. Taking Notes & Documenting Sources
• 7. Evaluating Sources
• 8. Citations & Plagiarism
• 9. Writing Your Research Paper

Evaluation Criteria

It's very important to evaluate the materials you find to make sure they are appropriate for a research paper.  It's not enough that the information is relevant; it must also be credible.  You will want to find more than enough resources, so that you can pick and choose the best for your paper.   Here are some helpful criteria you can apply to the information you find:

C urrency :

• When was the information published?
• Is the source out-of-date for the topic? 
• Are there new discoveries or important events since the publication date?

R elevancy:

• How is the information related to your argument? 
• Is the information too advanced or too simple? 
• Is the audience focus appropriate for a research paper? 
• Are there better sources elsewhere?

A uthority :

• Who is the author? 
• What is the author's credential in the related field? 
• Is the publisher well-known in the field? 
• Did the information go through the peer-review process or some kind of fact-checking?

A ccuracy :

• Can the information be verified? 
• Are sources cited? 
• Is the information factual or opinion based?
• Is the information biased? 
• Is the information free of grammatical or spelling errors?
• What is the motive of providing the information: to inform? to sell? to persuade? to entertain?
• Does the author or publisher make their intentions clear? Who is the intended audience?

Evaluating Web Sources

Most web pages are not fact-checked or anything like that, so it's especially important to evaluate information you find on the web.  Many articles on websites are fine for information, and many others are distorted or made up.  Check out our media evaluation guide for tips on evaluating what you see on social media, news sites, blogs, and so on.

This three-part video series, in which university students, historians, and pro fact-checkers go head-to-head in checking out online information, is also helpful.

• << Previous: 6. Taking Notes & Documenting Sources
• Next: 8. Citations & Plagiarism >>
• Last Updated: Apr 18, 2023 12:12 PM
• URL: https://butte.libguides.com/ResearchPaper

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Working with sources
• Evaluating Sources | Methods & Examples

Evaluating Sources | Methods & Examples

Published on June 2, 2022 by Eoghan Ryan . Revised on May 31, 2023.

The sources you use are an important component of your research. It’s important to evaluate the sources you’re considering using, in order to:

• Ensure that they’re credible
• Determine whether they’re relevant to your topic
• Assess the quality of their arguments

Table of contents

Evaluating a source’s credibility, evaluating a source’s relevance, evaluating a source’s arguments, other interesting articles, frequently asked questions about evaluating sources.

Evaluating the credibility of a source is an important way of sifting out misinformation and determining whether you should use it in your research. Useful approaches include the CRAAP test and lateral reading .

One of the best ways to evaluate source credibility is the CRAAP test . This stands for:

• Currency: Does the source reflect recent research?
• Relevance: Is the source related to your research topic?
• Authority: Is it a respected publication? Is the author an expert in their field?
• Accuracy: Does the source support its arguments and conclusions with evidence?
• Purpose: What is the author’s intention?

How you evaluate a source using these criteria will depend on your subject and focus. It’s important to understand the types of sources and how you should use them in your field of research.

Lateral reading

Lateral reading is the act of evaluating the credibility of a source by comparing it to other sources. This allows you to:

• Verify evidence
• Contextualize information
• Find potential weaknesses

If a source is using methods or drawing conclusions that are incompatible with other research in its field, it may not be reliable.

Rather than taking these figures at face value, you decide to determine the accuracy of the source’s claims by cross-checking them with official statistics such as census reports and figures compiled by the Department of Homeland Security’s Office of Immigration Statistics.

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

• Missing commas and periods
• Incorrect usage of “et al.”
• Ampersands (&) in narrative citations
• Missing reference entries

How you evaluate the relevance of a source will depend on your topic, and on where you are in the research process . Preliminary evaluation helps you to pick out relevant sources in your search, while in-depth evaluation allows you to understand how they’re related.

Preliminary evaluation

As you cannot possibly read every source related to your topic, you can use preliminary evaluation to determine which sources might be relevant. This is especially important when you’re surveying a large number of sources (e.g., in a literature review or systematic review ).

One way to do this is to look at paratextual material, or the parts of a work other than the text itself.

• Look at the table of contents to determine the scope of the work.
• Consult the index for key terms or the names of important scholars.

You can also read abstracts , prefaces , introductions , and conclusions . These will give you a clear idea of the author’s intentions, the parameters of the research, and even the conclusions they draw.

Preliminary evaluation is useful as it allows you to:

• Determine whether a source is worth examining in more depth
• Quickly move on to more relevant sources
• Increase the quality of the information you consume

While this preliminary evaluation is an important step in the research process, you should engage with sources more deeply in order to adequately understand them.

In-depth evaluation

Begin your in-depth evaluation with any landmark studies in your field of research, or with sources that you’re sure are related to your research topic.

As you read, try to understand the connections between the sources. Look for:

• Key debates: What topics or questions are currently influencing research? How does the source respond to these key debates?
• Major publications or critics: Are there any specific texts or scholars that have greatly influenced the field? How does the source engage with them?
• Trends: Is the field currently dominated by particular theories or research methods ? How does the source respond to these?
• Gaps: Are there any oversights or weaknesses in the research?

Even sources whose conclusions you disagree with can be relevant, as they can strengthen your argument by offering alternative perspectives.

Every source should contribute to the debate about its topic by taking a clear position. This position and the conclusions the author comes to should be supported by evidence from direct observation or from other sources.

Most sources will use a mix of primary and secondary sources to form an argument . It is important to consider how the author uses these sources. A good argument should be based on analysis and critique, and there should be a logical relationship between evidence and conclusions.

To assess an argument’s strengths and weaknesses, ask:

• Does the evidence support the claim?
• How does the author use evidence? What theories, methods, or models do they use?
• Could the evidence be used to draw other conclusions? Can it be interpreted differently?
• How does the author situate their argument in the field? Do they agree or disagree with other scholars? Do they confirm or challenge established knowledge?

Situating a source in relation to other sources ( lateral reading ) can help you determine whether the author’s arguments and conclusions are reliable and how you will respond to them in your own writing.

If you want to know more about ChatGPT, AI tools , citation , and plagiarism , make sure to check out some of our other articles with explanations and examples.

• ChatGPT vs human editor
• ChatGPT citations
• Is ChatGPT trustworthy?
• Using ChatGPT for your studies
• What is ChatGPT?
• Chicago style
• Paraphrasing

 Plagiarism

• Types of plagiarism
• Self-plagiarism
• Avoiding plagiarism
• Academic integrity
• Consequences of plagiarism
• Common knowledge

Prevent plagiarism. Run a free check.

As you cannot possibly read every source related to your topic, it’s important to evaluate sources to assess their relevance. Use preliminary evaluation to determine whether a source is worth examining in more depth.

This involves:

• Reading abstracts , prefaces, introductions , and conclusions
• Looking at the table of contents to determine the scope of the work
• Consulting the index for key terms or the names of important scholars

Lateral reading is the act of evaluating the credibility of a source by comparing it with other sources. This allows you to:

A credible source should pass the CRAAP test  and follow these guidelines:

• The information should be up to date and current.
• The author and publication should be a trusted authority on the subject you are researching.
• The sources the author cited should be easy to find, clear, and unbiased.
• For a web source, the URL and layout should signify that it is trustworthy.

The CRAAP test is an acronym to help you evaluate the credibility of a source you are considering using. It is an important component of information literacy .

The CRAAP test has five main components:

• Currency: Is the source up to date?
• Relevance: Is the source relevant to your research?
• Authority: Where is the source published? Who is the author? Are they considered reputable and trustworthy in their field?
• Accuracy: Is the source supported by evidence? Are the claims cited correctly?
• Purpose: What was the motive behind publishing this source?

Scholarly sources are written by experts in their field and are typically subjected to peer review . They are intended for a scholarly audience, include a full bibliography, and use scholarly or technical language. For these reasons, they are typically considered credible sources .

Popular sources like magazines and news articles are typically written by journalists. These types of sources usually don’t include a bibliography and are written for a popular, rather than academic, audience. They are not always reliable and may be written from a biased or uninformed perspective, but they can still be cited in some contexts.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Ryan, E. (2023, May 31). Evaluating Sources | Methods & Examples. Scribbr. Retrieved September 9, 2024, from https://www.scribbr.com/working-with-sources/evaluating-sources/

Is this article helpful?

Eoghan Ryan

Other students also liked, student guide: information literacy | meaning & examples, types of sources explained | examples & tips, what are credible sources & how to spot them | examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

12.7 Evaluation: Effectiveness of Research Paper

Learning outcomes.

By the end of this section, you will be able to:

• Identify common formats and design features for different kinds of texts.
• Implement style and language consistent with argumentative research writing while maintaining your own voice.
• Determine how genre conventions for structure, paragraphing, tone, and mechanics vary.

When drafting, you follow your strongest research interests and try to answer the question on which you have settled. However, sometimes what began as a paper about one thing becomes a paper about something else. Your peer review partner will have helped you identify any such issues and given you some insight regarding revision. Another strategy is to compare and contrast your draft with the grading rubric similar to one your instructor will use. It is a good idea to consult this rubric frequently throughout the drafting process.

This book may not be used in the training of large language models or otherwise be ingested into large language models or generative AI offerings without OpenStax's permission.

Want to cite, share, or modify this book? This book uses the Creative Commons Attribution License and you must attribute OpenStax.

Access for free at https://openstax.org/books/writing-guide/pages/1-unit-introduction

• Authors: Michelle Bachelor Robinson, Maria Jerskey, featuring Toby Fulwiler
• Publisher/website: OpenStax
• Book title: Writing Guide with Handbook
• Publication date: Dec 21, 2021
• Location: Houston, Texas
• Book URL: https://openstax.org/books/writing-guide/pages/1-unit-introduction
• Section URL: https://openstax.org/books/writing-guide/pages/12-7-evaluation-effectiveness-of-research-paper

© Dec 19, 2023 OpenStax. Textbook content produced by OpenStax is licensed under a Creative Commons Attribution License . The OpenStax name, OpenStax logo, OpenStax book covers, OpenStax CNX name, and OpenStax CNX logo are not subject to the Creative Commons license and may not be reproduced without the prior and express written consent of Rice University.

• The Open University
• Accessibility hub
• Guest user / Sign out
• Study with The Open University

My OpenLearn Profile

Personalise your OpenLearn profile, save your favourite content and get recognition for your learning

About this free course

Become an ou student, download this course, share this free course.

Start this free course now. Just create an account and sign in. Enrol and complete the course for a free statement of participation or digital badge if available.

1 Important points to consider when critically evaluating published research papers

Simple review articles (also referred to as ‘narrative’ or ‘selective’ reviews), systematic reviews and meta-analyses provide rapid overviews and ‘snapshots’ of progress made within a field, summarising a given topic or research area. They can serve as useful guides, or as current and comprehensive ‘sources’ of information, and can act as a point of reference to relevant primary research studies within a given scientific area. Narrative or systematic reviews are often used as a first step towards a more detailed investigation of a topic or a specific enquiry (a hypothesis or research question), or to establish critical awareness of a rapidly-moving field (you will be required to demonstrate this as part of an assignment, an essay or a dissertation at postgraduate level).

The majority of primary ‘empirical’ research papers essentially follow the same structure (abbreviated here as IMRAD). There is a section on Introduction, followed by the Methods, then the Results, which includes figures and tables showing data described in the paper, and a Discussion. The paper typically ends with a Conclusion, and References and Acknowledgements sections.

The Title of the paper provides a concise first impression. The Abstract follows the basic structure of the extended article. It provides an ‘accessible’ and concise summary of the aims, methods, results and conclusions. The Introduction provides useful background information and context, and typically outlines the aims and objectives of the study. The Abstract can serve as a useful summary of the paper, presenting the purpose, scope and major findings. However, simply reading the abstract alone is not a substitute for critically reading the whole article. To really get a good understanding and to be able to critically evaluate a research study, it is necessary to read on.

While most research papers follow the above format, variations do exist. For example, the results and discussion sections may be combined. In some journals the materials and methods may follow the discussion, and in two of the most widely read journals, Science and Nature, the format does vary from the above due to restrictions on the length of articles. In addition, there may be supporting documents that accompany a paper, including supplementary materials such as supporting data, tables, figures, videos and so on. There may also be commentaries or editorials associated with a topical research paper, which provide an overview or critique of the study being presented.

Box 1 Key questions to ask when appraising a research paper

• Is the study’s research question relevant?
• Does the study add anything new to current knowledge and understanding?
• Does the study test a stated hypothesis?
• Is the design of the study appropriate to the research question?
• Do the study methods address key potential sources of bias?
• Were suitable ‘controls’ included in the study?
• Were the statistical analyses appropriate and applied correctly?
• Is there a clear statement of findings?
• Does the data support the authors’ conclusions?
• Are there any conflicts of interest or ethical concerns?

There are various strategies used in reading a scientific research paper, and one of these is to start with the title and the abstract, then look at the figures and tables, and move on to the introduction, before turning to the results and discussion, and finally, interrogating the methods.

Another strategy (outlined below) is to begin with the abstract and then the discussion, take a look at the methods, and then the results section (including any relevant tables and figures), before moving on to look more closely at the discussion and, finally, the conclusion. You should choose a strategy that works best for you. However, asking the ‘right’ questions is a central feature of critical appraisal, as with any enquiry, so where should you begin? Here are some critical questions to consider when evaluating a research paper.

Look at the Abstract and then the Discussion : Are these accessible and of general relevance or are they detailed, with far-reaching conclusions? Is it clear why the study was undertaken? Why are the conclusions important? Does the study add anything new to current knowledge and understanding? The reasons why a particular study design or statistical method were chosen should also be clear from reading a research paper. What is the research question being asked? Does the study test a stated hypothesis? Is the design of the study appropriate to the research question? Have the authors considered the limitations of their study and have they discussed these in context?

Take a look at the Methods : Were there any practical difficulties that could have compromised the study or its implementation? Were these considered in the protocol? Were there any missing values and, if so, was the number of missing values too large to permit meaningful analysis? Was the number of samples (cases or participants) too small to establish meaningful significance? Do the study methods address key potential sources of bias? Were suitable ‘controls’ included in the study? If controls are missing or not appropriate to the study design, we cannot be confident that the results really show what is happening in an experiment. Were the statistical analyses appropriate and applied correctly? Do the authors point out the limitations of methods or tests used? Were the methods referenced and described in sufficient detail for others to repeat or extend the study?

Take a look at the Results section and relevant tables and figures : Is there a clear statement of findings? Were the results expected? Do they make sense? What data supports them? Do the tables and figures clearly describe the data (highlighting trends etc.)? Try to distinguish between what the data show and what the authors say they show (i.e. their interpretation).

Moving on to look in greater depth at the Discussion and Conclusion : Are the results discussed in relation to similar (previous) studies? Do the authors indulge in excessive speculation? Are limitations of the study adequately addressed? Were the objectives of the study met and the hypothesis supported or refuted (and is a clear explanation provided)? Does the data support the authors’ conclusions? Maybe there is only one experiment to support a point. More often, several different experiments or approaches combine to support a particular conclusion. A rule of thumb here is that if multiple approaches and multiple lines of evidence from different directions are presented, and all point to the same conclusion, then the conclusions are more credible. But do question all assumptions. Identify any implicit or hidden assumptions that the authors may have used when interpreting their data. Be wary of data that is mixed up with interpretation and speculation! Remember, just because it is published, does not mean that it is right.

O ther points you should consider when evaluating a research paper : Are there any financial, ethical or other conflicts of interest associated with the study, its authors and sponsors? Are there ethical concerns with the study itself? Looking at the references, consider if the authors have preferentially cited their own previous publications (i.e. needlessly), and whether the list of references are recent (ensuring that the analysis is up-to-date). Finally, from a practical perspective, you should move beyond the text of a research paper, talk to your peers about it, consult available commentaries, online links to references and other external sources to help clarify any aspects you don’t understand.

The above can be taken as a general guide to help you begin to critically evaluate a scientific research paper, but only in the broadest sense. Do bear in mind that the way that research evidence is critiqued will also differ slightly according to the type of study being appraised, whether observational or experimental, and each study will have additional aspects that would need to be evaluated separately. For criteria recommended for the evaluation of qualitative research papers, see the article by Mildred Blaxter (1996), available online. Details are in the References.

Activity 1 Critical appraisal of a scientific research paper

A critical appraisal checklist, which you can download via the link below, can act as a useful tool to help you to interrogate research papers. The checklist is divided into four sections, broadly covering:

• some general aspects
• research design and methodology
• the results
• discussion, conclusion and references.

Science perspective – critical appraisal checklist [ Tip: hold Ctrl and click a link to open it in a new tab. ( Hide tip ) ]

• Identify and obtain a research article based on a topic of your own choosing, using a search engine such as Google Scholar or PubMed (for example).
• The selection criteria for your target paper are as follows: the article must be an open access primary research paper (not a review) containing empirical data, published in the last 2–3 years, and preferably no more than 5–6 pages in length.
• Critically evaluate the research paper using the checklist provided, making notes on the key points and your overall impression.

Critical appraisal checklists are useful tools to help assess the quality of a study. Assessment of various factors, including the importance of the research question, the design and methodology of a study, the validity of the results and their usefulness (application or relevance), the legitimacy of the conclusions, and any potential conflicts of interest, are an important part of the critical appraisal process. Limitations and further improvements can then be considered.

• Library Home
• Research Guides

Writing a Research Paper

• Evaluate Sources

Library Research Guide

• Choose Your Topic
• Organize Your Information
• Draft Your Paper
• Revise, Review, Refine

How Will This Help Me?

Evaluating your sources will help you:

• Determine the credibility of information
• Rule out questionable information
• Check for bias in your sources

In general, websites are hosted in domains that tell you what type of site it is.

• .com = commercial
• .net = network provider
• .org = organization
• .edu = education
• .mil = military
• .gov = U.S. government

Commercial sites want to persuade you to buy something, and organizations may want to persuade you to see an issue from a particular viewpoint. 

Useful information can be found on all kinds of sites, but you must consider carefully whether the source is useful for your purpose and for your audience.

Content Farms

Content farms are websites that exist to host ads. They post about popular web searches to try to drive traffic to their sites. They are rarely good sources for research.

• Web’s “Content Farms” Grow Audiences For Ads This article by Zoe Chace at National Public Radio describes the ways How To sites try to drive more traffic to their sites to see the ads they host.

Fact Checking

Fact checking can help you verify the reliability of a source. The following sites may not have all the answers, but they can help you look into the sources for statements made in U.S. politics.

• FactCheck.org This site monitors the accuracy of statements made in speeches, debates, interviews, and more and links to sources so readers can see the information for themselves. The site is a project of the Annenberg Public Policy Center of the University of Pennsylvania.
• PolitiFact This resource evaluates the accuracy of statements made by elected officials, lobbyists, and special interest groups and provides sources for their evaluations. PolitiFact is currently run by the nonprofit Poynter Institute for Media Studies.

Evaluate Sources With the Big 5 Criteria

The Big 5 Criteria can help you evaluate your sources for credibility:

• Currency: Check the publication date and determine whether it is sufficiently current for your topic.
• Coverage (relevance): Consider whether the source is relevant to your research and whether it covers the topic adequately for your needs.
• Authority: Discover the credentials of the authors of the source and determine their level of expertise and knowledge about the subject.
• Accuracy: Consider whether the source presents accurate information and whether you can verify that information. 
• Objectivity (purpose): Think about the author's purpose in creating the source and consider how that affects its usefulness to your research. 

Evaluate Sources With the CRAAP Test

Another way to evaluate your sources is the CRAAP Test, which means evaluating the following qualities of your sources:

This video (2:17) from Western Libraries explains the CRAAP Test. 

Video transcript

Evaluating Sources ( Western Libraries ) CC BY-NC-ND 3.0

Evaluate Websites

Evaluating websites follows the same process as for other sources, but finding the information you need to make an assessment can be more challenging with websites. The following guidelines can help you decide if a website is a good choice for a source for your paper. 

• Currency . A useful site is updated regularly and lets visitors know when content was published on the site. Can you tell when the site was last updated? Can you see when the content you need was added? Does the site show signs of not being maintained (broken links, out-of-date information, etc.)?
• Relevance . Think about the target audience for the site. Is it appropriate for you or your paper's audience?
• Authority . Look for an About Us link or something similar to learn about the site's creator. The more you know about the credentials and mission of a site's creators, as well as their sources of information, the better idea you will have about the site's quality. 
• Accuracy. Does the site present references or links to the sources of information it presents? Can you locate these sources so that you can read and interpret the information yourself?
• Purpose. Consider the reason why the site was created. Can you detect any bias? Does the site use emotional language? Is the site trying to persuade you about something? 

Identify Political Perspective

News outlets, think tanks, organizations, and individual authors can present information from a particular political perspective. Consider this fact to help determine whether sources are useful for your paper. 

Check a news outlet's website, usually under About Us or Contact Us , for information about their reporters and authors. For example, USA Today has the USA Today Reporter Index , and the LA Times has an Editorial & Newsroom Contacts . Reading a profile or bio for a reporter or looking at other articles by the author may tell you whether that person favors a particular viewpoint. 

If a particular organization is mentioned in an article, learn more about the organization to identify potential biases. Think tanks and other associations usually exist for a reason. Searching news articles about the organization can help you determine their political leaning. 

Bias is not always bad, but you must be aware of it. Knowing the perspective of a source helps contextualize the information presented. 

• << Previous: Databases
• Next: Organize Your Information >>
• Last Updated: Sep 4, 2024 2:42 PM
• URL: https://guides.lib.k-state.edu/writingresearchpaper

K-State Libraries

1117 Mid-Campus Drive North, Manhattan, KS 66506

785-532-3014 | [email protected]

• Statements and Disclosures
• Accessibility
• © Kansas State University

• Your Science & Health Librarians
• How To Find Articles with Databases
• Video Learning
• Artificial Intelligence Tools
• Industry Reports
• How To Evaluate Articles
• Search Tips, General
• Develop a Research Question
• How To Read A Scientific Paper
• How To Interpret Data
• How To Write A Scientific Paper
• Teaching Materials
• Systematic & Evideced-Based Reviews
• Get More Help
• New Books, STEM areas

Useful Sources

• How to Critically Evaluate the Quality of a Research Article?
• How to Evaluate Journal Articles

Evaluating Articles

Not all articles are created equally. Evaluating sources for relevancy and usefulness is one the most important steps in research. This helps researchers in the STEM disciplines to gather the information they need. There are several tools to use when evaluating an article. 

• Purposes can include persuasion, informing, or proving something to the reader. Depending on the topic of your paper, you will want to evaluate the article's purpose to support your own position. 

Publication

• Most sources for college papers should come from scholarly journals. Scholarly journals are journals that are peer-reviewed before publication. This means that experts in the field read and approve the material in the article before it is published.
• The date of publication is especially relevant to those in the STEM disciplines. Research in science, technology, engineering, and mathematics moves very quickly, so have an article that was published recently will be more useful. 
• Many universities operate their own press, Indiana University included! Books and articles that are published by a reputable institution are generally accepted as valuable information. Other notable presses included Harvard University Press and The MIT Press. 
• Good sources of information come from experts in the field. This is called authority. Many times these individuals will be employed at research institutions such as universities, labs, or founded associations. 
• Many times, authors with authority have written more than one article about a topic in their field. Not only does this add support to their reputation, but can also be a great source for more articles. 
• Citation is a great indicator to the effectiveness of the article. If other experts are citing the article, it is a good sign this source is trusted and relevant. 

Bibliography

• Scholarly works will always contain a bibliography of the sources used. Trusted articles will have sources that are also scholarly in nature and authored by individuals with authority in their field. 
• Much like evaluating the publication of an article, the bibliography of the source should also contain sources that are up-to-date. 
• << Previous: Industry Reports
• Next: Search Tips, General >>
• Last Updated: Sep 3, 2024 1:34 PM
• URL: https://guides.libraries.indiana.edu/STEM

Social media

• Instagram for Herman B Wells Library
• Facebook for IU Libraries

Additional resources

Featured databases.

• Resource available to authorized IU Bloomington users (on or off campus) OneSearch@IU
• Resource available to authorized IU Bloomington users (on or off campus) Academic Search (EBSCO)
• Resource available to authorized IU Bloomington users (on or off campus) ERIC (EBSCO)
• Resource available to authorized IU Bloomington users (on or off campus) Nexis Uni
• Resource available without restriction HathiTrust Digital Library
• Databases A-Z
• Resource available to authorized IU Bloomington users (on or off campus) Google Scholar
• Resource available to authorized IU Bloomington users (on or off campus) JSTOR
• Resource available to authorized IU Bloomington users (on or off campus) Web of Science
• Resource available to authorized IU Bloomington users (on or off campus) Scopus
• Resource available to authorized IU Bloomington users (on or off campus) WorldCat

IU Libraries

• Diversity Resources
• About IU Libraries
• Alumni & Friends
• Departments & Staff
• Jobs & Libraries HR
• Intranet (Staff)
• IUL site admin

• Systematic review
• Open access
• Published: 11 September 2024

Evaluation of research co-design in health: a systematic overview of reviews and development of a framework

• Sanne Peters   ORCID: orcid.org/0000-0001-6235-1752 1 ,
• Lisa Guccione 2 , 3 ,
• Jill Francis 1 , 2 , 3 , 4 ,
• Stephanie Best 1 , 2 , 3 , 5 ,
• Emma Tavender 6 , 7 ,
• Janet Curran 8 , 9 ,
• Katie Davies 10 ,
• Stephanie Rowe 1 , 8 ,
• Victoria J. Palmer 11 &
• Marlena Klaic 1  

Implementation Science volume  19 , Article number:  63 ( 2024 ) Cite this article

13 Altmetric

Metrics details

Co-design with consumers and healthcare professionals is widely used in applied health research. While this approach appears to be ethically the right thing to do, a rigorous evaluation of its process and impact is frequently missing. Evaluation of research co-design is important to identify areas of improvement in the methods and processes, as well as to determine whether research co-design leads to better outcomes. We aimed to build on current literature to develop a framework to assist researchers with the evaluation of co-design processes and impacts.

A multifaceted, iterative approach, including three steps, was undertaken to develop a Co-design Evaluation Framework: 1) A systematic overview of reviews; 2) Stakeholder panel meetings to discuss and debate findings from the overview of reviews and 3) Consensus meeting with stakeholder panel. The systematic overview of reviews included relevant papers published between 2000 and 2022. OVID (Medline, Embase, PsycINFO), EBSCOhost (Cinahl) and the Cochrane Database of Systematic reviews were searched for papers that reported co-design evaluation or outcomes in health research. Extracted data was inductively analysed and evaluation themes were identified. Review findings were presented to a stakeholder panel, including consumers, healthcare professionals and researchers, to interpret and critique. A consensus meeting, including a nominal group technique, was applied to agree upon the Co-design Evaluation Framework.

A total of 51 reviews were included in the systematic overview of reviews. Fifteen evaluation themes were identified and grouped into the following seven clusters: People (within co-design group), group processes, research processes, co-design context, people (outside co-design group), system and sustainment. If evaluation methods were mentioned, they mainly included qualitative data, informal consumer feedback and researchers’ reflections. The Co-Design Evaluation Framework used a tree metaphor to represent the processes and people in the co-design group (below-ground), underpinning system- and people-level outcomes beyond the co-design group (above-ground). To evaluate research co-design, researchers may wish to consider any or all components in the tree.

Conclusions

The Co-Design Evaluation Framework has been collaboratively developed with various stakeholders to be used prospectively (planning for evaluation), concurrently (making adjustments during the co-design process) and retrospectively (reviewing past co-design efforts to inform future activities).

Peer Review reports

Contributions to the literature

While stakeholder engagement in research seems ethically the right thing to do, a rigorous evaluation of its process and outcomes is frequently missing.

Fifteen evaluation themes were identified in the literature, of which research process , cognitive and emotional factors were the most frequently reported.

The Co-design Evaluation Framework can assist researchers with research co-design evaluation and provide guidance regarding what and when to evaluate.

The framework can be used prospectively, concurrently, and retrospectively to make improvements to existing and future research co-design projects.

Introduction

Lots of money is wasted in health research that does not lead to meaningful benefits for end-users, such as healthcare professionals and consumers [ 1 , 2 , 3 ]. One contributor to this waste is that research often focusses on questions and outcomes that are of limited importance to end-users [ 4 , 5 ]. Engaging relevant people in research co-design has increased in order to respond to this issue. There is a lack of consensus in the literature on the definition and processes involved in undertaking a co-design approach. For the purposes of this review, we define research co-design as meaningful end-user engagement that occurs across any stage of the research process , from the research planning phase to dissemination of research findings [ 6 ]. Meaningful end-user engagement refers to an explicit and measurable responsibility, such as contributing to writing a study proposal [ 6 ]. The variety of research co-design methods can be seen as a continuum ranging from limited involvement, such as consulting with end-users, to the much higher effort research approaches in which end-users and researchers aim for equal decision-making power and responsibility across the entire research process [ 6 ]. Irrespective of the intensity of involvement, it is generally recommended that a co-design approach should be based on several important principles such as equity, inclusion and shared ownership [ 7 ].

Over time, increasing attention has been given to research co-design [ 6 , 8 ]. Funding bodies encourage its use and it is recommended in the updated UK MRC framework on developing and evaluating complex interventions [ 9 ]. End-user engagement has an Equator reporting checklist [ 10 ] and related work has been reported by key organisations, such as the James Lind Alliance in the UK ( www.jla.nihr.ac.uk ), Patient Centered Outcomes Research Institute in the US ( www.pcori.org ) and Canadian Institutes of Health Research ( https://cihrirsc.gc.ca/e/41592.html ). In addition, peer reviewed publications involving co-design have risen from 173 per year in 2000 to 2617 in 2022 (PubMed), suggesting a growing importance in research activities.

Engaging end-users in the health research process is arguably the right thing to do, but the processes and outcomes of co-design have rarely been evaluated in a rigorous way [ 6 ]. Existing anecdotal evidence suggests that research co-design can benefit researchers, end-users and lead to more robust research processes [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ]. Both researchers and end-users have reported positive experiences of engaging in the co-design process. Potential benefits include a better understanding of community needs, more applicable research questions, designs and materials and improved trust between the researchers and end-users. Several reviews on conducting research co-design have concluded that co-design can be feasible, though predominantly used in the early phases of research, for example formulating research questions and developing a study protocol [ 6 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ]. However, these reviews highlighted that engagement of end-users in the research process required extra time and funding and had the risk of becoming tokenistic [ 6 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ].

The use of resources in co-design studies might need to be justified to the funder as well as its impacts. A rigorous evaluation of research co-design processes and outcomes is needed to identify areas of potential improvement and to determine the impact of research co-design. Several overviews of reviews on research co-design have been published but with no or limited focus on evaluation [ 20 , 21 , 22 , 23 ]. Moreover, current literature provides little guidance around how and what to evaluate, and which outcomes are key.

This study thus had two aims:

To conduct a systematic overview of reviews to identify evaluation methods and process and outcome variables reported in the published health research co-design literature.

To develop a framework to assist researchers with the evaluation of co-design processes and impacts.

This project used a multifaceted, iterative approach to develop a Co-design Evaluation Framework. It consisted of the following steps: 1) A systematic overview of reviews; 2) Stakeholder panel meetings to discuss and debate findings from the overview of reviews and 3) Consensus meeting with stakeholder panel. The reporting checklist for overviews of reviews was applied in Additional file 1 [ 24 ].

Step 1: A systematic overview of reviews

We conducted a systematic overview of reviews [ 25 ], reviewing literature reviews rather than primary studies, to investigate the following question: What is known in the published literature about the evaluation of research co-design in health research? The protocol of our systematic overview of reviews was published in the PROSPERO database (CRD42022355338).

Sub questions:

What has been co-designed and what were the objectives of the co-design process?

Who was involved and what was the level of involvement?

What methods were used to evaluate the co-design processes and outcomes?

What was evaluated (outcome and process measures) and at what timepoint (for example concurrently, or after, the co-design process)?

Was a co-design evaluation framework used to guide evaluation?

Search strategy

We searched OVID (Medline, Embase, PsycINFO), EBSCOhost (Cinahl) and the Cochrane Database of Systematic reviews on the 11th of October 2022 for literature reviews that reported co-design evaluation or outcomes in health research. The search strategy was based on previous reviews on co-design [ 6 , 14 , 26 ] and refined with the assistance of a research librarian and the research team (search terms in Additional file 2). Papers published from January 2000 to September 2022 were identified and retrieved by one author (SP).

Study selection

Database records were imported into EndNote X9 (The EndNote Team, Philadelphia, 2013) and duplicates removed. We managed the study selection process in the software program Covidence (Veritas Health Innovation, Melbourne, Australia). Two independent reviewers (SP, MK or LG) screened the titles and abstracts of all studies against the eligibility criteria (Table  1 ). Discrepancies were resolved through discussion or with a third reviewer (either SP, MK or LG, depending on which 2 reviewers disagreed). If there was insufficient information in the abstract to decide about eligibility, the paper was retained to the full-text screening phase. Full-text versions of studies not excluded at the title and abstract screening phase were retrieved and independently screened by two reviewers (SP, MK or LG) against eligibility criteria. Disagreements were resolved through discussion, or with a third reviewer, and recorded in Covidence.

Data extraction of included papers was conducted by one of three reviewers (SP, MK or LG). A second reviewer checked a random sample of 20% of all extracted data (LG or SP). Disagreements were resolved through regular discussion. Data were extracted using an excel spreadsheet developed by the research team and included review characteristics (such as references, type of review, number of included studies, review aim), details about the co-design process (such as who was involved in the co-design, which topics the co-design focused on, what research phase(s) the co-design covered, in which research phase the co-design took place and what the end-users’ level of involvement was) and details about the co-design evaluation (what outcomes were reported, methods of data collection, who the participants of the evaluation were, the timepoint of evaluation, whether an evaluation framework was used or developed and conclusions about co-design evaluation).

Types of end-users’ involvement were categorised into four groups based on the categories proposed by Hughes et al. (2018): 1. Targeted consultation; 2. Embedded consultation; 3. Collaboration and co-production and 4. User-led research, see Table  2 .

Data extraction and analysis took place in three iterative phases (Fig.  1 ), with each phase containing one third of the included studies. Each phase of data extraction and analysis was followed by stakeholder panel meetings (see step 2 below). This stepwise approach enabled a form of triangulation wherein themes that emerged through each phase were discussed with the stakeholder panel and incorporated both retrospectively (re-coding data in the prior phase) and prospectively (coding new data in the next phase).

Iterative phases in the process of the Co-design evaluation framework development

All reported outcomes of research co-design in the first phase (one third of all data) were inductively coded into themes, according to the principles of thematic analysis [ 28 ]. Two researchers (SP and MK) double coded 10% of all data and reached consensus through discussion. Given that consensus was high, one researcher (SP) continued the coding while having frequent discussions and reviews within the research team. In phase 2 (also one third of all data), deductive coding was based on the themes identified in the first round. Data of the first phase were re-coded, if new codes emerged during the stakeholder panel meeting. The same process took place for the third phase.

Step 2: Stakeholder panel meetings to discuss and debate findings from the overview of reviews

Results from step 1 were presented to the stakeholder panel to interpret and critique the review findings. The panel consisted of ten people, including a mix of consumers, healthcare professionals and researchers. Stakeholders were selected for their experience or expertise in research co-design. The number of meetings was not pre-determined, rather, it was informed by the outcomes from step 1. The number of stakeholders in each meeting ranged from six to ten.

A core group from the broader stakeholder panel (SP, MK, LG, JF) with a breadth of research experience and methodological expertise discussed the themes arising from both steps 1 and 2 and considered various ways of presenting them. Multiple design options were considered and preliminary frameworks were developed. Following discussion with the stakeholder panel, it was agreed that the evaluation themes could be grouped into several clusters to make the framework more comprehensible. The grouping of evaluation themes into clusters was informed by reported proposed associations between evaluation themes in the literature as well as the stakeholder panel’s co-design experience and expertise. Evaluation themes as well as clusters were agreed upon during the stakeholder panel meetings.

Step 3: Consensus meeting with stakeholder panel

The consensus meeting included the same stakeholder panel as in step 2. The meeting was informed by a modified Nominal Group Technique (NGT). The NGT is a structured process for obtaining information and reaching consensus with a target group who have some association or experience with the topic [ 29 ]. Various adaptations of the NGT have been used and additional pre-meeting information has been suggested to enable more time for participants to consider their contribution to the topic [ 30 ]. The modified NGT utilised in this study contained the following: (i) identification of group members to include experts with depth and diverse experiences. They were purposively identified at the start of this study for their expertise or experience in research co-design and included: a patient consumer, a clinician, three clinician researchers and six researchers with backgrounds in behavioural sciences, psychology, education, applied ethics and participatory design. All authors on this paper were invited by e-mail to attend an online meeting; (ii) provision of information prior to the group meeting included findings of the overview of reviews, a draft framework and objectives of the meeting. Five authors with extensive research co-design experience were asked to prepare a case example of one of their co-design projects for sharing at the group meeting. The intention of this exercise was to discuss the fit between a real-world example and the proposed framework; (iii) hybrid meeting facilitated by two researchers (SP & JF) who have experience in facilitating consensus meetings. Following presentation of the meeting materials, including the preliminary framework, group members were invited to silently consider the preliminary framework and generate ideas and critiques; iv) participants sharing their ideas and critiques; v) clarification process where group members shared their co-design example project and discussed the fit with components of the initial framework, and vi) silent voting and/or agreement on the framework via a personal email to one of the researchers (SP).

Step 1: Systematic overview of reviews

The database searches identified a total of 8912 papers. After removing 3016 duplicates and screening 5896 titles and abstracts, 148 full texts were sought for retrieval. Sixteen were not retrieved as they were not available in English ( n  = 2) or full-text was not available ( n  = 14). Of the remaining 132 papers assessed for eligibility, 81 were excluded. The final number of papers included in this overview of reviews was 51 (See Fig.  2 ).

PRISMA flow chart (based on [ 31 ]) of overview of reviews

Characteristics of the included studies

Of the 51 included reviews [ 11 , 12 , 14 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 ], 17 were systematic reviews, 12 were scoping reviews, 14 did not report the type or method of review, three were narrative reviews, two were qualitative evidence synthesis, another two were a structured literature search and one was a realist review. The number of studies included in the reviews ranged from 7 to 260. Nineteen reviews focused on co-design with specific populations, for example older people, people with intellectual disabilities, people living with dementia and 32 reviews included co-design with a range of end-users. The co-design focused in most cases on a mix of topics ( n  = 31). Some reviews were specifically about one clinical topic, for example critical care or dementia. In ten cases, the clinical topics were not reported. Co-design took place during multiple research phases. Thirty-six reviews covered co-design in agenda/priority setting, 36 in study design, 30 in data collection, 25 in data analysis and 27 in dissemination. With regards to the research translation continuum, most of the co-design was reported in practice and community-based research ( n  = 32), three reviews were conducted in basic research and 11 in human research. The types of end-users’ involvement in co-design ranged from targeted consultation ( n  = 14) to embedded consultation ( n  = 20), collaboration and co-production ( n  = 14) to end-user- led research ( n  = 6), including papers covering multiple types of involvement. Seventeen papers did not report the types of involvement. The reported co-design included a variety of time commitments, from a minimum of a one-off 60-min meeting to multiple meetings over multiple years. Twenty-seven reviews did not report details about the end-users’ types of involvement.

Identified evaluation themes

Fifteen evaluation themes were identified and were arranged into two higher level groups: 1. within the co-design team and 2. broader than co-design team (Table  3 ). The themes related to the first group (within the co-design team) included: Structure and composition of the co-design group, contextual enablers/barriers, interrelationships between group members, decision making process, emotional factors, cognitive factors, value proposition, level/ quality of engagement, research process, health outcomes for co-design group and sustainment of the co-design team or activities. The themes within the second group (broader than co-design team) included: Healthcare professional-level outcomes, healthcare system level outcomes, organisational level outcomes and patient and community outcomes.

The research process was the most frequently reported evaluation theme in the reviews ( n  = 44, 86% of reviews), followed by cognitive factors ( n  = 35, 69%) and emotional factors ( n  = 34, 67%) (Table  4 ). Due to variability in reporting practices, it was not possible to specify the number of primary studies that reported specific evaluation themes. Evaluation methods for the themes were not reported in the majority of reviews ( n  = 43, 84%). If evaluation methods were mentioned, they were mainly based on qualitative data, including interviews, focus groups, field notes, document reviews and observations (see overview with references in Additional file 3). Survey data was mentioned in three reviews. Many reviews reported informal evaluation based on participant experiences (e.g. informal feedback), reflection meetings, narrative reflections and authors’ hypotheses (Additional file 3). The timing of the evaluation was only mentioned in two papers: 1. Before and after the co-design activities and 2. Post co-design activities. One paper suggested that continuous evaluation might be helpful to improve the co-design process (Additional file 3).

The systematic overview of reviews found that some authors reported proposed positive associations between evaluation themes (Table  5 ). The most frequently reported proposed association was between level/quality of engagement and emotional factors ( n  = 5, 10%). However, these proposed associations did not seem to have any empirical evidence and evaluation methods were not reported.

All evaluation themes were grouped into the following clusters (Table  6 ): People (within co-design group), group processes, research processes, co-design context, people (outside co-design group), system and sustainment.

Only one paper reported the evaluation in connection to the research phases (Agenda/priority setting, study design, data collection, data analysis and dissemination). This paper reported the following outcomes for the following research phases [ 58 ]:

Agenda/priority setting: Research process; Level/quality of engagement; Cognitive factors; Attributes of the co-design group; Interrelationships between group members; Sustainment of the co-design team or activities; Patient and community outcomes.

Study design: Attributes of the co-design group; Interrelationships between group members; Level/quality of engagement; Cognitive factors; Emotional factors; Research process.

The various research phases in which consumers could be involved, as well as the clusters of evaluation themes, informed the design of the co-design evaluation framework.

Two main options were voted on and discussed within the stakeholder panel. The two main options can be found in Additional file 4. Draft 2 was the prefered option as it was perceived as more dynamic than draft 1, representing a clearer interplay between the two contexts. The stakeholder panel suggested a few edits to the draft, such as the inclusion of bi-directional arrows in the tree trunk and a vertical arrow from underground to above ground with the label ‘impact’.

The final version of the Co-design Evaluation framework is presented in Fig.  3 .

Research Co-design evaluation framework

Figure  3 presents co-design evaluation as the below-ground and above-ground structures of a tree. The tree metaphor presents the processes and people in the co-design group (below-ground) as the basis for system- and people-level outcomes beyond the co-design group (above-ground). To evaluate research co-design, researchers may wish to consider any or all components in this Figure. These evaluation components relate to the methods, processes, and outcomes of consumer involvement in research.

The context within the co-design group (the roots of the tree) consists of the people, group processes and research processes, with various evaluation themes (dot points) related to them, as well as contextual barriers and enablers that relate to situational aspects that might enable or hinder consumer engagement. The context outside the co-design group, i.e., the wider community (the branches and leaves of the tree), comprises people who were not involved in the research co-design process, the system-level and sustainment-related outcomes. These above ground groups are potential beneficiaries or targets of the co-design activities.

The arrows in the middle of the trunk represent the potential mutual influence of the two contexts, suggesting that an iterative approach to evaluation might be beneficial. For example, when deciding the composition of the co-design group, it may be important to have an appropriate representation of the people most impacted by the problem issue or topic at hand. Or, if a co-designed healthcare intervention does not achieve the desired outcomes in the wider context, the co-design group might consider potential ways to improve the intervention or how it was delivered. Evaluation of a research co-design process might start with the foundations (the roots of the tree) and progress to above ground (the tree grows and might develop fruit). Yet, depending on the aim of the evaluation, a focus on one of the two contexts, either below or above ground, might be appropriate.

Which, and how many, components are appropriate to evaluate depends on the nature of the co-design approach and the key questions of the evaluation. For example, if a co-design approach is used in the very early stages of a research program, perhaps to identify priorities or to articulate a research question, then 'below' the ground components are key. While a randomised study comparing the effects of a co-designed intervention versus a researcher-designed intervention might only consider 'above' the ground components.

The white boxes on the right-hand side of Fig.  3 indicate the research phases, from agenda/priority setting to dissemination, in which consumers can and should be involved. This co-design evaluation framework may be applied at any phase of the research process or applied iteratively with a view to improving future co-design activities.

This systematic overview of reviews aimed to build on current literature and develop a framework to assist researchers with the evaluation of research co-design. Fifty-one included reviews reported on fifteen evaluation themes, which were grouped into the following clusters: People (within co-design group), group processes, research processes, co-design context, people (outside co-design group), system and sustainment. Most reviews did not report measurement methods for the evaluation themes. If methods were mentioned, they mostly included qualitative data, informal consumer feedback and researchers’ reflections. This finding strengthens our argument that a framework may be helpful in supporting methodologically robust studies to assess co-design processes and impacts. The Co-Design Evaluation Framework has adopted a tree metaphor. It presents the processes and people in the co-design group (below-ground) as the underpinning system- and people-level outcomes beyond the co-design group (above-ground). To evaluate stakeholder involvement in research, researchers may wish to consider any or all components in the tree. Which, and how many, components are appropriate to evaluate depends on the nature of the co-design approach and the key questions that stakeholders aim to address. Nonetheless, it will be important that evaluations delineate what parts of the research project have incorporated a co-design approach.

The Equator reporting checklist for Research Co-Design, GRIPP2, provides researchers with a series of concepts that should be considered and reported on when incorporating patient and public involvement in research [ 10 ]. These concepts include, but are not limited to, methods of involving patients and the public in research and intensity of engagement. The Co-Design Evaluation Framework is not intended as a replacement for the GRIPP2, rather, it can be used prospectively to inform development of the co-design project or retropsectively to inform completion of the GRIPP2. Table 7 provides hypothetical examples of research questions that co-design evaluation projects might address. The framework could be used at multiple points within co-design projects, including prospectively (planning for evaluation before the co-design process has started), concurrently ( incorporating improvements during the co-design process) and retrospectively (reviewing past co-design efforts to inform future projects).

Our systematic overview of reviews identified multiple evaluation themes. Some of these overlapped with reported values associated with public involvement in research [ 80 ], community engagement measures [ 15 ] and reported impacts of patient and public involvement in research, as described by others [ 16 , 81 , 82 ]. The added value of our systematic overview of reviews is that we went beyond a list of items and took it one step further by looking at evaluation themes, potential associations between evaluation themes, clusters of evaluation themes and ultimately developed a framework to assist others with research co-design evaluation.

Some reviews in our overview of reviews proposed potential associations between evaluation themes. Yet, these proposed associations were not empirically tested. One of the included studies [ 58 ] proposed conditions and mechanisms involved in co-design processes and outcomes related to diabetes research. Although it is a promising starting point, this should be further explored. A realist evaluation including other research topics and other approaches, such as the use of logic models, which was also recognised in the updated MRC framework [ 9 ], might help to build on explorations of included mechanisms of action [ 83 ] and give insight into how core ingredients contribute to certain co-design processes and outcomes. As recognised by others [ 6 , 84 ], the reporting practice of research co-design in the literature could be improved as details about context, mechanisms and expected outcomes are frequently missing. This will also help us to gain a better understanding of what works for whom, why, how and in which circumstances.

The lack of a consistent definition of co-design makes it challenging to identify and synthesise the literature, as recognised by others [ 6 ]. Given that there are so many different terms used in the literature, there is a risk that we might have missed some relevant papers in our overview of reviews. Nevertheless, we tried to capture as many as possible synonyms of co-design in our search terms. The absence of quality assessment of included studies in our overview of reviews can be seen as a limitation. However, our overview of reviews did not aim to assess existing literature on the co-design process, but rather focused on what to evaluate, how and when. We did note whether the reported evaluation themes were based on empirical evidence or authors’ opinions. Primary studies reported in the included reviews were not individually reviewed as this was outside the scope of this paper. A strength in our methods was the cyclical process undertaken between steps 1 and 2. Analysis of the data extracted from the overview was refined over three phases following rigorous discussions with a diverse and experienced stakeholder panel. It was a strength of our project that a mix of stakeholders were involved, including consumers, healthcare professionals and researchers.

Stakeholders are frequently engaged in research but if research co-design processes and outcomes are not evaluated, there will be limited learning from past experiences. Evaluation is essential to make refinements during existing projects and improve future co-design activities. It is also critical for ensuring commitments to the underpinning values of c-odesign are embedded within activities.

A systematic review of all primary studies within the included reviews of this overview of reviews, would allow greater depth relating to the practicalities of how to evaluate certain themes. It would lead to a better understanding of existing measures and methods and which evaluation areas need further development. Future research should also focus on whether co-design leads to better outcomes than no co-design (only researcher-driven research). To our knowledge, this has not been explored yet. Moreover, future research could gain better insight into the mechanisms of change within co-design and explore potential associations between evaluation themes for example, those proposed in the included reviews between level/quality of engagement and emotional factors.

We followed a systematic, iterative approach to develop a Co-Design Evaluation Framework that can be applied to various phases of the research co-design process. Testing of the utility of the framework is an important next step. We propose that the framework could be used at multiple points within co-design projects, including prospectively (planning for evaluation before the co-design process has started), concurrently (to incorporate improvements during the co-design process) and retrospectively (reviewing past co-design efforts to inform future projects).

Availability of data and materials

All data generated during this study are included either within the text or as a supplementary file.

Abbreviations

Medical Research Council

Guidance for Reporting Involvement of Patients and the Public

Chalmers I, Glasziou P. Systematic reviews and research waste. Lancet. 2016;387(10014):122–3.

Article   PubMed   Google Scholar  

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76.

Ioannidis JP. Why Most Clinical Research Is Not Useful. PLoS Med. 2016;13(6):e1002049.

Article   PubMed   PubMed Central   Google Scholar  

Oliver S. Patient involvement in setting research agendas. Eur J Gastroenterol Hepatol. 2006;18(9):935–8.

Slattery P, Saeri AK, Bragge P. Research co-design in health: a rapid overview of reviews. Health Res Policy Syst. 2020;18(1):17.

Ní Shé É, Harrison R. Mitigating unintended consequences of co-design in health care. Health Expect. 2021;24(5):1551–6.

Peters S, Sukumar K, Blanchard S, Ramasamy A, Malinowski J, Ginex P, et al. Trends in guideline implementation: an updated scoping review. Implement Sci. 2022;17:50.

Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374:n2061.

Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14:89.

Manafo E, Petermann L, Mason-Lai P, Vandall-Walker V. Patient engagement in Canada: a scoping review of the “how” and “what” of patient engagement in health research. Health Res Policy Syst. 2018;16(1):5.

Fergusson D, Monfaredi Z, Pussegoda K, Garritty C, Lyddiatt A, Shea B, et al. The prevalence of patient engagement in published trials: a systematic review. Res Involv Engagem. 2018;4:17.

Vat LE, Finlay T, Jan Schuitmaker-Warnaar T, Fahy N, Robinson P, Boudes M, et al. Evaluating the “return on patient engagement initiatives” in medicines research and development: A literature review. Health Expect. 2020;23(1):5–18.

Luger TM, Hamilton AB, True G. Measuring Community-Engaged Research Contexts, Processes, and Outcomes: A Mapping Review. Milbank Q. 2020;98(2):493–553.

Modigh A, Sampaio F, Moberg L, Fredriksson M. The impact of patient and public involvement in health research versus healthcare: A scoping review of reviews. Health Policy. 2021;125(9):1208–21.

Clavel N, Paquette J, Dumez V, Del Grande C, Ghadiri DPS, Pomey MP, et al. Patient engagement in care: A scoping review of recently validated tools assessing patients’ and healthcare professionals’ preferences and experience. Health Expect. 2021;24(6):1924–35.

Newman B, Joseph K, Chauhan A, Seale H, Li J, Manias E, et al. Do patient engagement interventions work for all patients? A systematic review and realist synthesis of interventions to enhance patient safety. Health Expect. 2021;24:1905 No Pagination Specified.

Lowe D, Ryan R, Schonfeld L, Merner B, Walsh L, Graham-Wisener L, et al. Effects of consumers and health providers working in partnership on health services planning, delivery and evaluation. Cochrane Database Syst Rev. 2021;9:CD013373.

PubMed   Google Scholar  

Price A, Albarqouni L, Kirkpatrick J, Clarke M, Liew SM, Roberts N, et al. Patient and public involvement in the design of clinical trials: An overview of systematic reviews. J Eval Clin Pract. 2018;24(1):240–53.

Sarrami-Foroushani P, Travaglia J, Debono D, Braithwaite J. Implementing strategies in consumer and community engagement in health care: results of a large-scale, scoping meta-review. BMC Health Serv Res. 2014;14:402.

Abrams R, Park S, Wong G, Rastogi J, Boylan A-M, Tierney S, et al. Lost in reviews: Looking for the involvement of stakeholders, patients, public and other non-researcher contributors in realist reviews. Research Synthesis Methods. 2021;12(2):239–47.

Zych MM, Berta WB, Gagliardi AR. Conceptualising the initiation of researcher and research user partnerships: a meta-narrative review. Health Res Policy Syst. 2020;18(1):24.

Gates M, Gates A, Pieper D, Fernandes RM, Tricco AC, Moher D, et al. Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. BMJ. 2022;378:e070849.

Pollock A, Campbell P, Brunton G, Hunt H, Estcourt L. Selecting and implementing overview methods: implications from five exemplar overviews. Syst Rev. 2017;6(1):145.

Greenhalgh T, Hinton L, Finlay T, Macfarlane A, Fahy N, Clyde B, et al. Frameworks for supporting patient and public involvement in research: Systematic review and co-design pilot. Health Expectations: An International Journal of Public Participation in Health Care & Health Policy. 2019;22(4):785–801.

Article   Google Scholar  

Hughes M, Duffy C. Public involvement in health and social sciences research: A concept analysis. Health Expectations: An International Journal of Public Participation in Health Care & Health Policy. 2018;21(6):1183–90.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2008;3(2):77–101.

Waggoner J, Carline JD, Durning SJ. Is There a Consensus on Consensus Methodology? Descriptions and Recommendations for Future Consensus Research. Acad Med. 2016;91(5):663–8.

Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18(2):188–94.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Baldwin JN, Napier S, Neville S, Clair VAWS. Impacts of older people’s patient and public involvement in health and social care research: a systematic review. Age Ageing. 2018;47(6):801–9.

Bench S, Eassom E, Poursanidou K. The nature and extent of service user involvement in critical care research and quality improvement: A scoping review of the literature. Int J Consum Stud. 2018;42(2):217–31.

Bethell J, Commisso E, Rostad HM, Puts M, Babineau J, Grinbergs-Saull A, et al. Patient engagement in research related to dementia: a scoping review. Dementia. 2018;17(8):944–75.

Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. The Patient: Patient-Centered Outcomes Research. 2014;7(4):387–95.

Di Lorito C, Birt L, Poland F, Csipke E, Gove D, Diaz-Ponce A, et al. A synthesis of the evidence on peer research with potentially vulnerable adults: how this relates to dementia. Int J Geriatr Psychiatry. 2017;32(1):58–67.

Di Lorito C, Bosco A, Birt L, Hassiotis A. Co-research with adults with intellectual disability: A systematic review. J Appl Res Intellect Disabil. 2018;31(5):669–86.

Fox G, Fergusson DA, Daham Z, Youssef M, Foster M, Poole E, et al. Patient engagement in preclinical laboratory research: A scoping review. EBioMedicine. 2021;70:103484.

Frankena TK, Naaldenberg J, Cardol M, Linehan C, van Schrojenstein Lantman-de Valk H. Active involvement of people with intellectual disabilities in health research - a structured literature review. Res Dev Disabil. 2015;45–46:271–83.

Fudge N, Wolfe CD, McKevitt C. Involving older people in health research. Age Ageing. 2007;36(5):492–500.

Article   CAS   PubMed   Google Scholar  

George AS, Mehra V, Scott K, Sriram V. Community participation in health systems research: A systematic review assessing the state of research, the nature of interventions involved and the features of engagement with communities. PLoS One. 2015;10(10):ArtID e0141091.

Legare F, Boivin A, van der Weijden T, Pakenham C, Burgers J, Legare J, et al. Patient and public involvement in clinical practice guidelines: A knowledge synthesis of existing programs. Med Decis Making. 2011;31(6):E45–74.

McCarron TL, Clement F, Rasiah J, Moran C, Moffat K, Gonzalez A, et al. Patients as partners in health research: a scoping review. Health Expect. 2021;24:1378 No Pagination Specified.

Miller J, Knott V, Wilson C, Roder D. A review of community engagement in cancer control studies among indigenous people of Australia, New Zealand, Canada and the USA. Eur J Cancer Care. 2012;21(3):283–95.

Article   CAS   Google Scholar  

Shen S, Doyle-Thomas KAR, Beesley L, Karmali A, Williams L, Tanel N, et al. How and why should we engage parents as co-researchers in health research? A scoping review of current practices. Health Expect. 2017;20(4):543–54.

Velvin G, Hartman T, Bathen T. Patient involvement in rare diseases research: a scoping review of the literature and mixed method evaluation of Norwegian researchers’ experiences and perceptions. Orphanet J Rare Dis. 2022;17(1):212.

Wiles LK, Kay D, Luker JA, Worley A, Austin J, Ball A, et al. Consumer engagement in health care policy, research and services: A systematic review and meta-analysis of methods and effects. PLoS One. 2022;17:e0261808.no pagination

Cook N, Siddiqi N, Twiddy M, Kenyon R. Patient and public involvement in health research in low and middle-income countries: a systematic review. BMJ Open. 2019;9(5):e026514.

Chambers E, Gardiner C, Thompson J, Seymour J. Patient and carer involvement in palliative care research: An integrative qualitative evidence synthesis review. Palliat Med. 2019;33(8):969–84.

Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17(5):637–50.

Boaz A, Hanney S, Jones T, Soper B. Does the engagement of clinicians and organisations in research improve healthcare performance: a three-stage review. BMJ Open. 2015;5(12):e009415.

Boivin A, L’Esperance A, Gauvin FP, Dumez V, Macaulay AC, Lehoux P, et al. Patient and public engagement in research and health system decision making: A systematic review of evaluation tools. Health Expect. 2018;21(6):1075–84.

Anderst A, Conroy K, Fairbrother G, Hallam L, McPhail A, Taylor V. Engaging consumers in health research: a narrative review. Aust Health Rev. 2020;44(5):806–13.

Arnstein L, Wadsworth AC, Yamamoto BA, Stephens R, Sehmi K, Jones R, et al. Patient involvement in preparing health research peer-reviewed publications or results summaries: a systematic review and evidence-based recommendations. Res Involv Engagem. 2020;6:34.

Becerril-Montekio V, Garcia-Bello LA, Torres-Pereda P, Alcalde-Rabanal J, Reveiz L, Langlois EV. Collaboration between health system decision makers and professional researchers to coproduce knowledge, a scoping review. Int J Health Plann Manage. 2022;28:28.

Google Scholar  

Bird M, Ouellette C, Whitmore C, Li L, Nair K, McGillion MH, et al. Preparing for patient partnership: A scoping review of patient partner engagement and evaluation in research. Health Expect. 2020;23:523 No Pagination Specified.

Dawson S, Campbell SM, Giles SJ, Morris RL, Cheraghi-Sohi S. Black and minority ethnic group involvement in health and social care research: A systematic review. Health Expect. 2018;21(1):3–22.

Harris J, Haltbakk J, Dunning T, Austrheim G, Kirkevold M, Johnson M, et al. How patient and community involvement in diabetes research influences health outcomes: A realist review. Health Expectations: An International Journal of Public Participation in Health Care & Health Policy. 2019;22(5):907–20.

Hubbard G, Kidd L, Donaghy E. Involving people affected by cancer in research: a review of literature. Eur J Cancer Care. 2008;17(3):233–44.

Hubbard G, Kidd L, Donaghy E, McDonald C, Kearney N. A review of literature about involving people affected by cancer in research, policy and planning and practice. Patient Educ Couns. 2007;65(1):21–33.

Jones EL, Williams-Yesson BA, Hackett RC, Staniszewska SH, Evans D, Francis NK. Quality of reporting on patient and public involvement within surgical research: a systematic review. Ann Surg. 2015;261(2):243–50.

Drahota A, Meza RD, Brikho B, Naaf M, Estabillo JA, Gomez ED, et al. Community-Academic Partnerships: A Systematic Review of the State of the Literature and Recommendations for Future Research. Milbank Q. 2016;94(1):163–214.

Forsythe LP, Szydlowski V, Murad MH, Ip S, Wang Z, Elraiyah TA, et al. A systematic review of approaches for engaging patients for research on rare diseases. J Gen Intern Med. 2014;29(Suppl 3):788–800.

Article   PubMed Central   Google Scholar  

Lander J, Hainz T, Hirschberg I, Strech D. Current practice of public involvement activities in biomedical research and innovation: a systematic qualitative review. PLoS One [Electronic Resource]. 2014;9(12):e113274.

Lee DJ, Avulova S, Conwill R, Barocas DA. Patient engagement in the design and execution of urologic oncology research. Urol Oncol. 2017;35(9):552–8.

Malterud K, Elvbakken KT. Patients participating as co-researchers in health research: A systematic review of outcomes and experiences. Scandinavian Journal of Public Health. 2020;48(6):617–28.

Miah J, Dawes P, Edwards S, Leroi I, Starling B, Parsons S. Patient and public involvement in dementia research in the European Union: a scoping review. BMC geriatr. 2019;19(1):220.

Nilsen ES, Myrhaug HT, Johansen M, Oliver S, Oxman AD. Methods of consumer involvement in developing healthcare policy and research, clinical practice guidelines and patient information material. Cochrane Database Syst Rev. 2006;2006(3):CD004563.

PubMed   PubMed Central   Google Scholar  

Oliver S, Clarke-Jones L, Rees R, Milne R, Buchanan P, Gabbay J, et al. Involving consumers in research and development agenda setting for the NHS: developing an evidence-based approach. Health Technol Assess. 2004;8(15):1-148 III-IV.

Orlowski SK, Lawn S, Venning A, Winsall M, Jones GM, Wyld K, et al. Participatory Research as One Piece of the Puzzle: A Systematic Review of Consumer Involvement in Design of Technology-Based Youth Mental Health and Well-Being Interventions. JMIR Hum Factors. 2015;2(2):e12.

Pii KH, Schou LH, Piil K, Jarden M. Current trends in patient and public involvement in cancer research: A systematic review. Health Expectations: An International Journal of Public Participation in Health Care & Health Policy. 2019;22(1):3–20.

Sandoval JA, Lucero J, Oetzel J, Avila M, Belone L, Mau M, et al. Process and outcome constructs for evaluating community-based participatory research projects: a matrix of existing measures. Health Educ Res. 2012;27(4):680–90.

Sangill C, Buus N, Hybholt L, Berring LL. Service user’s actual involvement in mental health research practices: A scoping review. Int J Ment Health Nurs. 2019;28(4):798–815.

Schelven F, Boeije H, Marien V, Rademakers J. Patient and public involvement of young people with a chronic condition in projects in health and social care: A scoping review. Health Expect. 2020;23:789 No Pagination Specified.

Schilling I, Gerhardus A. Methods for Involving Older People in Health Research-A Review of the Literature. International Journal of Environmental Research & Public Health [Electronic Resource]. 2017;14(12):29.

Shippee ND, Domecq Garces JP, Prutsky Lopez GJ, Wang Z, Elraiyah TA, Nabhan M, et al. Patient and service user engagement in research: a systematic review and synthesized framework. Health Expect. 2015;18(5):1151–66.

Vaughn LM, Jacquez F, Lindquist-Grantz R, Parsons A, Melink K. Immigrants as research partners: A review of immigrants in community-based participatory research (CBPR). J Immigr Minor Health. 2017;19(6):1457–68.

Walmsley J, Strnadova I, Johnson K. The added value of inclusive research. J Appl Res Intellect Disabil. 2018;31(5):751–9.

Weschke S, Franzen DL, Sierawska AK, Bonde LS, Strech D. Schorr SG. Reporting of patient involvement: A mixed-methods analysis of current practice in health research publications. medRxiv; 2022. p. 21.

Gradinger F, Britten N, Wyatt K, Froggatt K, Gibson A, Jacoby A, et al. Values associated with public involvement in health and social care research: A narrative review. Health Expectations: An International Journal of Public Participation in Health Care & Health Policy. 2015;18(5):661–75.

Hoekstra F, Mrklas KJ, Khan M, McKay RC, Vis-Dunbar M, Sibley KM, et al. A review of reviews on principles, strategies, outcomes and impacts of research partnerships approaches: A first step in synthesising the research partnership literature. Health Res Pol Syst. 2020;18(1):51 no pagination.

Stallings SC, Boyer AP, Joosten YA, Novak LL, Richmond A, Vaughn YC, et al. A taxonomy of impacts on clinical and translational research from community stakeholder engagement. Health Expect. 2019;22(4):731–42.

Grindell C, Coates E, Croot L, O’Cathain A. The use of co-production, co-design and co-creation to mobilise knowledge in the management of health conditions: a systematic review. BMC Health Serv Res. 2022;22(1):877.

Staley K. “Is it worth doing?” Measuring the impact of patient and public involvement in research. Res Involv Engagem. 2015;1:6.

Download references

Acknowledgements

The authors would like to thank the graphic designers, Jenni Quinn and Kevin Calthorpe, for their work on Fig. 3 .

Not applicable.

Author information

Authors and affiliations.

School of Health Sciences, The University of Melbourne, Melbourne, Australia

Sanne Peters, Jill Francis, Stephanie Best, Stephanie Rowe & Marlena Klaic

Department of Health Services Research, Peter MacCallum Cancer Centre, Melbourne, Australia

Lisa Guccione, Jill Francis & Stephanie Best

Sir Peter MacCallum Department of Oncology, Faculty of Medicine, Dentistry and Health Sciences, The University of Melbourne, Melbourne, Australia

Centre for Implementation Research, Ottawa Hospital Research Institute, Ottawa, Canada

Jill Francis

Victorian Comprehensive Cancer Centre, Melbourne, VIC, Australia

Stephanie Best

Emergency Research, Murdoch Children’s Research Institute, Melbourne, Australia

Emma Tavender

Department of Critical Care, The University of Melbourne , Melbourne, Australia

School of Nursing, Faculty of Health, Ottawa, Canada

Janet Curran & Stephanie Rowe

Emergency Medicine, Faculty of Medicine, Ottawa, Canada

Janet Curran

Neurological Rehabilitation Group Mount Waverley, Mount Waverley, Australia

Katie Davies

The ALIVE National Centre for Mental Health Research Translation, The University of Melbourne, Melbourne, Australia

Victoria J. Palmer

You can also search for this author in PubMed   Google Scholar

Contributions

SP coordinated the authorship team, completed the systematic literature searches, synthesis of data, framework design and substantial writing. MK and LG were the second reviewers for the systematic overview of reviews. MK, LG and JF assisted with framework design.  SP, LG, JF, SB, ET, JC, KD, SR, VP and MK participated in the stakeholder meetings and the consensus process. All authors commented on drafts and approved the final submitted version of the manuscript.

Corresponding author

Correspondence to Sanne Peters .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., supplementary material 4., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Peters, S., Guccione, L., Francis, J. et al. Evaluation of research co-design in health: a systematic overview of reviews and development of a framework. Implementation Sci 19 , 63 (2024). https://doi.org/10.1186/s13012-024-01394-4

Download citation

Received : 01 April 2024

Accepted : 31 August 2024

Published : 11 September 2024

DOI : https://doi.org/10.1186/s13012-024-01394-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research co-design
• Stakeholder involvement
• End-user engagement
• Consumer participation
• Outcome measures

Implementation Science

ISSN: 1748-5908

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• CAREER FEATURE
• 09 September 2024

The human costs of the research-assessment culture

• Rachel Brazil 0

Rachel Brazil is a freelance journalist in London, UK.

You can also search for this author in PubMed   Google Scholar

Illustration: Stephan Schmitz

You have full access to this article via your institution.

The term ‘REF-able’ is now in common usage in UK universities. “Everyone’s constantly thinking of research in terms of ‘REF-able’ outputs, in terms of ‘REF-able’ impact,” says Richard Watermeyer, a sociologist at the University of Bristol, UK. He is referring to the UK Research Excellence Framework (REF), which is meant to happen every seven years and is one of the most intensive systems of academic evaluation in any country. “Its influence is ubiquitous — you can’t escape it,” says Watermeyer. But he and other scholars around the world are concerned about the effects of an extreme audit culture in higher education, one in which researchers’ productivity is continually measured and, in the case of the REF, directly tied to research funding for institutions. Critics say that such systems are having a detrimental effect on staff and, in some cases, are damaging researchers’ mental health and departmental collegiality.

Unlike other research benchmarking systems, the REF results directly affect the distribution of around £2 billion (US$2.6 billion) annually, creating high stakes for institutions. UK universities receive a significant proportion of their government funding in this way (in addition to the research grants awarded to individual academics).

Research assessment toolkit

Since its inception, the REF methodology has been through several iterations. The rules about which individuals’ work must be highlighted have changed, but there has always been a focus on peer-review panels to assess outputs. Since 2014, a team in each university department has been tasked with selecting a dossier of research outputs and case studies that must demonstrate societal impact. These submissions can receive anything from a four-star rating (for the most important, world-leading research) to just one star (the least significant work, of only national interest). Most departments aim to include three- or four-star submissions, often described as ‘REF-able’.

But the process is time-consuming and does not come cheap. The most recent REF, in 2021, was estimated to have cost £471 million. Tanita Casci, director of the Research Strategy & Policy Unit at the University of Oxford, UK, acknowledges that it’s resource-intensive, but says that it’s still a very efficient way of distributing funds, compared with the cost of allocating money through individual grant proposals. “I don’t think the alternative is better,” she concludes. The next exercise has been pushed back a year, until 2029, with planned changes to include a larger emphasis on assessment of institutional research culture.

Tanita Casci says the UK REF assessment is an efficient way to distribute funding. Credit: University of Oxford

Many UK academics see the REF as adding to an already highly competitive and stressful environment. A 2021 survey of more than 3,000 researchers (see go.nature.com/47umnjd ) found that they generally felt that the burdens of the REF outweighed the benefits. They also thought that it had decreased academics’ ability to follow their own intellectual interests and disincentivized the pursuit of riskier, more-speculative work with unpredictable outcomes.

Some other countries have joined the assessment train — with the notable exception of the United States, where the federal government does not typically award universities general-purpose research funding. But no nation has chosen to copy the REF exactly. Some, such as the Netherlands, have instead developed a model that challenges departments to set their own strategic goals and provide evidence that they have achieved them.

Whatever the system, few assessments loom as large in the academic consciousness as the REF. “You will encounter some institutions where, if you mention the REF, there’s a sort of groan and people talk about how stressed it’s making them,” says Petra Boynton, a research consultant and former health-care researcher at University College London.

Strain on team spirit

Staff collating a department’s REF submission, selecting the research outputs and case studies to illustrate impact, can find themselves in an uncomfortable position, says Watermeyer. He was involved in his own department’s 2014 submission and has published a study of the REF’s emotional toll 1 . It’s a job that most academics take on “with trepidation”, he says. It can change how they interact with colleagues and how colleagues view and interact with them.

“You’re trying to make robust, dispassionate, critical determinations of the quality of research. Yet at the back of your mind, you are inescapably aware of the implications of the judgements that you’re making in terms of people’s research identities, their careers,” says Watermeyer. In his experience, people can get quite defensive. That scrutiny of close colleagues’ work “can be really disruptive and damaging to relationships”.

UK research assessment is being reformed — but the changes miss the mark

Watermeyer often found himself not only adjudicating on work but also acting as a counsellor. “You have to attend to the emotional labour that’s involved; you’re responsible for people’s welfare and well-being,” and no training is provided, he says. A colleague might think that their work has met expectations, only to find that assessors disagree. “I’ve been in situations where there are tears,” Watermeyer recalls. “People break down.”

For university support staff, the REF also looms large. Sometimes, more staff must be hired near the submission deadline to cope with the workload. “It is an unbelievable pressure cooker,” particularly at small institutions, says Julie Bayley, former director of research-impact development at the University of Lincoln, UK. Bayley was responsible for overseeing 50 case studies to demonstrate the impact of Lincoln’s research, and describes this as akin to preparing evidence for a legal case. “You are having to prove, to a good level of scrutiny, that this claim is true,” Bayley says. This usually involves collecting testimonial letters from organizations or individuals who can vouch for the research impact, something she sometimes did on behalf of researchers who feared straining the external relationships they had developed.

Boynton says there can be an upside. “There’s something really exciting about putting together [a case study] that shows you did something amazing,” she says. But she also acknowledges that those whose research is not put forward can feel as if their work doesn’t matter or is not respected, and that can be demoralizing.

The clamour about achieving four stars can skew attitudes about research achievements. Bayley recounts a senior academic tearfully showing her an e-mail from his supervisor that read, “It’s all well and good that you’ve changed national UK policy, but unless you change European policy, it doesn’t count.” She says her own previous research on teenage pregnancy met with similar responses because it involved meeting real needs at the grass-roots level, rather than focusing on national policy. “That’s the bit I find most heartbreaking. Four-star is glory for the university, but four-star is not impact for society,” says Bayley.

The picking and choosing between individual researchers has implications for departments. “That places some people on the ‘star player competition winner’ side and, particularly where resources are limited, that means those people get more support” from their departments, explains Bayley. She has witnessed others being asked to pick up the teaching workload of researchers who are selected to produce impact case studies for a REF submission. Boynton agrees: “It’s not a collegiate, collective thing — it’s divisive.”

Hidden contributions

Research assessment can also affect work that universities often consider ‘non-REF-able’. Simon Hettrick, a research software engineer at the University of Southampton, UK, was in this position in 2021. He collaborates with researchers to produce crucial software for their work. But, he says, universities find it hard to look beyond academic papers as the metric for success even though there are 21 categories of research output that can be considered, including software, patents, conference proceedings and digital and visual media.

In the 2021 REF, publications made up about 98.5% of submissions. Hettrick says that although other submissions are encouraged, universities tend not to select the alternatives, presumably out of habit or for fear they might not be judged as favourably.

Simon Hettrick says evaluations should include more contributions such as software. Credit: Simon Hettrick

The result is that those in roles similar to Hettrick’s feel demotivated. “You’re working really hard, without the recognition for that input you’re making,” he says. To counter this, Hettrick and others launched an initiative called The hidden REF that ran a 2021 competition to spotlight important work unrecognized by the REF, garnering 120 submissions from more than 60 universities. The competition is being run again this year .

In April, Hettrick and his colleagues wrote a manifesto asking universities to ensure that at least 5% of their submissions for the 2029 REF are ‘non-traditional outputs’. “That has been met with some consternation,” he says.

Regarding career advancement, REF submissions should not feed into someone’s prospects, according to Casci, who says that universities make strong efforts to separate REF assessments from decisions about individuals’ career progression. But “it’s a grey area” in Watermeyer’s experience; “it might not be reflected within formal promotional criteria, but I think it’s the accepted unspoken reality”. He thinks that academic researchers lacking ‘REF-able’ three- or four-star outputs are unlikely to be hired by any “serious research institution” — severely limiting their career prospects and mobility.

Watermeyer says the consequences for these individuals will vary. Some institutions try to boost the ratings of early-career academics by putting them on capacity-building programmes, including buddying schemes to foster collaborations with more ‘REF-able’ colleagues. But, for more senior staff, the downside could be a performance review. “People might be ‘encouraged’ to reconsider their research role, if they find themselves unable to satisfy the three-star criteria,” he says.

There’s a similar imperative for a researcher’s work to be used as an impact case study. “If your work is not selected for that competition, you lose the currency for your own progression,” says Bayley.

The REF also exacerbates inequalities that already exist in research, says Emily Yarrow, an organizational-behaviour researcher at Newcastle University Business School, UK. “There are still gendered impacts and gendered effects of the REF, and still a disproportionate negative impact on those who take time out of their careers, for example, for caring responsibilities, maternity leave.” A 2014 analysis she co-authored of REF impact case studies in the fields of business and management showed that women were under-represented: just 25% of studies with an identifiable lead author were led by women 2 . Boynton also points out that there are clear inequalities in the resources available to institutions to prepare for the REF, causing many researchers to feel that the system is unfair.

Emily Yarrow found that women were under-represented in research-evaluation case studies. Credit: Toby Long

Although not all the problems researchers face can be attributed to the REF, it certainly contributes to what some have called an epidemic of poor mental health among UK higher-education staff. A 2019 report (see go.nature.com/3xsb78x ) highlighted the REF as causing administrative overload for some and evoking a heightened, ever-present fear of ‘failure’ for others.

UK research councils have acknowledged the criticisms and have promised changes to the 2029 REF. Steven Hill, chair of the 2021 REF Steering Group at Research England in Bristol, UK, which manages the REF exercise, says these changes will “rebalance the exercise’s definition of research excellence, to focus more on the environment needed for all talented people to thrive”. Hill also says they will implement changes to break “the link between individuals and submissions” because there will no longer be a minimum or maximum number of submissions for each researcher. The steering group aims to provide more support in terms of how REF guidance is applied by institutions, to dispel misconceptions about requirements. “Some institutions frame their performance criteria in REF terms and place greater requirements on staff than are actually required by REF,” Hill says.

Other ways forward

Similar to the REF, the China Discipline Evaluation (CDE) occurs every four to five years. Yiran Zhou, a higher-education researcher at the University of Cambridge, UK, has studied attitudes to the CDE 3 and says there are pressures in China to produce the equivalent of ‘REF-able’ research and similar concerns about the impact on academics. China relies much more on conventional quantitative publication metrics, but researchers Zhou interviewed criticized the time wasted in producing CDE impact case studies. Those tasked with organizing this often had to bargain with colleagues to collect the evidence they needed. “Then, they owe personal favours to them, like teaching for one or two hours,” says Zhou.

Increased competition has become a concern among Chinese universities, and Zhou says the government has decided not to publicize the results of the most recent CDE, only informing the individual universities. And, Zhou says, some of those she spoke to favoured dropping the assessment altogether.

Mammoth UK research assessment concludes as leaders eye radical shake up

In 2022, Australia did just that. Ahead of the country’s 2023 Excellence in Research for Australia (ERA) assessment, the government announced that it would stop the time-consuming process and start a transition to examine other “modern data-driven approaches, informed by expert review”. In October 2023, the Australian Research Council revealed a blueprint for a new assessment system and was investigating methods for smarter harvesting of evaluation data. It also noted that any data used would be “curated”, possibly with the help of artificial intelligence.

Some European countries are moving away from the type of competitive process exemplified by the REF. “For the Netherlands, we hope to move from evaluation to development” of careers and departmental strategies, says Kim Huijpen, programme manager for Recognition and Reward for the Universities of the Netherlands, based in The Hague, and a former chair of the working group of the Strategy Evaluation Protocol (SEP), the research evaluation process for Dutch universities. In the SEP, institutions organize subject-based research-unit evaluations every six years, but the outcome is not linked to government funding.

The SEP is a benchmarking process. Each research group selects indicators and other types of evidence related to its strategy and these, along with a site visit, provide the basis for review by a committee of peers and stakeholders. The protocol for 2021–27 has removed the previous system of grading. “We wanted to get away from this kind of ranking exercise,” explains Huijpen. “There’s a lot of freedom to deepen the conversation on quality, the societal relevance and the impact of the work — and it’s not very strict in how you should do this.”

The Research Council of Norway also runs subject-based assessments every decade, including institutional-level metrics and case studies, to broadly survey a field. “From what I hear from colleagues, the Norwegian assessment is much milder than the REF. Although it’s similar in what is looked at, it doesn’t feel the same,” says Alexander Refsum Jensenius, a music researcher at the University of Oslo. That’s probably because there is no direct link between the assessment and funding.

Refsum Jensenius has been involved in the Norwegian Career Assessment Matrix , a toolbox developed in 2021 by Universities Norway, the cooperative body of 32 accredited universities. It isn’t used to assess departments, but it demonstrates a fresh, broader approach.

What differentiates it from many other assessments is that in addition to providing evidence, there is scope for a researcher to outline the motivations for their research directions and make their own value judgements on achievements. “You cannot only have endless lists of whatever you have been doing, but you also need to reflect on it and perhaps suggest that some of these things have more value to you,” says Refsum Jensenius. For example, researchers might add context to their publication list by highlighting that opportunities to publish their work are limited by its interdisciplinary nature. There is also an element of continuing professional development to identify a researcher’s skills that need strengthening. Refsum Jensenius says this approach has been welcomed in the Norwegian system. “The toolbox is starting to be adopted by many institutions, including the University of Oslo, for hiring and promoting people.”

For many UK researchers, this more nurturing, reflective method of assessment might feel a million miles away from the REF, but that’s not to say that the REF process does not address ways to improve an institution’s research environment. Currently, one of the three pillars of assessment involves ‘people, culture and environment’, which includes open science, research integrity, career development and equity, diversity and inclusion (EDI) concerns. Since 2022, there have been discussions on how to better measure and incentivize good practice in these areas for the next REF.

Bayley thinks the REF can already take some credit for an increased emphasis on EDI issues at UK universities. “I will not pretend for a second it’s sorted, but EDI is now so commonly a standing item on agendas that it’s far more present than it ever was.”

But she is less sure that the REF has improved research culture overall. For example, she says after the 2014 REF, when the rules changed to require that contributions from all permanent research staff be submitted, she saw indications that some universities were gaming the system in a way that disadvantaged early-career researchers. Junior staff members were left on precarious temporary contracts, and she has seen examples of institutions freezing staff numbers to avoid the need to submit more impact case studies. “I’ve seen that many times across many universities, which means the early-career entry points for research roles are reduced.”

“The REF is a double-edged sword,” concludes Bayley. The administrative burden and pressures it brings are much too high, but it does provide a way to allocate money that gives smaller institutions more of a chance, she says. After the 2021 REF, even though top universities still dominated, many received less of the pot than previously, whereas some newer, less prestigious universities performed strongly. The biggest increase was at Northumbria University in Newcastle, where ‘quality-related’ funding rose from £7 million to £18 million.

For Watermeyer, the whole process is counterproductive, wasting precious resources and creating a competitive, rather than a collaborative, culture that might not tolerate the most creative thinkers. He would like to see it abolished. Hettrick is in two minds, because “the realist in me says it is necessary to explain to the taxpayer what we’re doing with their money”. He says the task now is to do the assessment more cheaply and more effectively.

Other research communities might not agree. As Huijpen points out, “there’s quite a lot of assessments in academic life, there are a lot of moments within a career where you are assessed, when you apply for funding, when you apply for a job”. From her perspective, it’s time to opt for less ranking and more reflection.

Nature 633 , 481-484 (2024)

doi: https://doi.org/10.1038/d41586-024-02922-4

Watermeyer, R., Derrick, G. E. & Batalla, M. B. Res. Eval. 31 , 498–506 (2022).

Article   Google Scholar  

Davies, J., Yarrow, E. & Syed, J. Gend. Work Organ. 27 , 129–148 (2020).

Zhou, Y. High. Educ. 88 , 1019–1035 (2024).

Download references

Related Articles

• Research management
• Institutions
• Peer review

The grassroots organizations continuing the fight for Ukrainian science

Career Feature 11 SEP 24

How a struggling biotech company became a university ‘spin-in’

Career Q&A 10 SEP 24

Massive Attack’s science-led drive to lower music’s carbon footprint

Career Feature 04 SEP 24

Why I’m committed to breaking the bias in large language models

Career Guide 04 SEP 24

Binning out-of-date chemicals? Somebody think about the carbon!

Correspondence 27 AUG 24

No more hunting for replication studies: crowdsourced database makes them easy to find

Nature Index 27 AUG 24

Can South Korea regain its edge in innovation?

Nature Index 21 AUG 24

What will it take to open South Korean research to the world?

How South Korea can support female research leaders

Open Rank Tenure-track/Tenured Faculty Positions in the Department of Genetic and Genomic Sciences

Seeking expertise in areas including Cancer Genetics; Artificial Intelligence; Drug Development & Clinical Trials; Functional Genomics; & Gene Editing

New York City, New York (US)

Mount Sinai Department of Genetics and Genomic Sciences

Assistant Professor (Tenure Track) of Robotics

The Department of Mechanical and Process Engineering (D-MAVT, www.mavt.ethz.ch) at ETH Zurich invites applications for the above-mentioned position.

Zurich city

Gathering Outstanding Overseas Talents, Innovating to Lead the Future

The 16th Peiyang Young Scientist Forum and the 2024 Tianjin University High-Level Forum for University Faculty, Postdoctoral Fellows...

Tianjin, China

Tianjin University (TJU)

High-level Talent Recruitment dedicated to teaching & research

College of Water Sciences, Beijing Normal University

Beijing, China

OSU Neurology Clayton C. Wagner Parkinson’s Disease Research Professorship

Columbus, Ohio

The Ohio State University (OSU)

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

This week: the arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Computation and Language

Title: a dataset for evaluating llm-based evaluation functions for research question extraction task.

Abstract: The progress in text summarization techniques has been remarkable. However the task of accurately extracting and summarizing necessary information from highly specialized documents such as research papers has not been sufficiently investigated. We are focusing on the task of extracting research questions (RQ) from research papers and construct a new dataset consisting of machine learning papers, RQ extracted from these papers by GPT-4, and human evaluations of the extracted RQ from multiple perspectives. Using this dataset, we systematically compared recently proposed LLM-based evaluation functions for summarizations, and found that none of the functions showed sufficiently high correlations with human evaluations. We expect our dataset provides a foundation for further research on developing better evaluation functions tailored to the RQ extraction task, and contribute to enhance the performance of the task. The dataset is available at this https URL .

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Clin Diagn Res
• v.11(5); 2017 May

Critical Appraisal of Clinical Research

Azzam al-jundi.

1 Professor, Department of Orthodontics, King Saud bin Abdul Aziz University for Health Sciences-College of Dentistry, Riyadh, Kingdom of Saudi Arabia.

Salah Sakka

2 Associate Professor, Department of Oral and Maxillofacial Surgery, Al Farabi Dental College, Riyadh, KSA.

Evidence-based practice is the integration of individual clinical expertise with the best available external clinical evidence from systematic research and patient’s values and expectations into the decision making process for patient care. It is a fundamental skill to be able to identify and appraise the best available evidence in order to integrate it with your own clinical experience and patients values. The aim of this article is to provide a robust and simple process for assessing the credibility of articles and their value to your clinical practice.

Introduction

Decisions related to patient value and care is carefully made following an essential process of integration of the best existing evidence, clinical experience and patient preference. Critical appraisal is the course of action for watchfully and systematically examining research to assess its reliability, value and relevance in order to direct professionals in their vital clinical decision making [ 1 ].

Critical appraisal is essential to:

• Combat information overload;
• Identify papers that are clinically relevant;
• Continuing Professional Development (CPD).

Carrying out Critical Appraisal:

Assessing the research methods used in the study is a prime step in its critical appraisal. This is done using checklists which are specific to the study design.

Standard Common Questions:

• What is the research question?
• What is the study type (design)?
• Selection issues.
• What are the outcome factors and how are they measured?
• What are the study factors and how are they measured?
• What important potential confounders are considered?
• What is the statistical method used in the study?
• Statistical results.
• What conclusions did the authors reach about the research question?
• Are ethical issues considered?

The Critical Appraisal starts by double checking the following main sections:

I. Overview of the paper:

• The publishing journal and the year
• The article title: Does it state key trial objectives?
• The author (s) and their institution (s)

The presence of a peer review process in journal acceptance protocols also adds robustness to the assessment criteria for research papers and hence would indicate a reduced likelihood of publication of poor quality research. Other areas to consider may include authors’ declarations of interest and potential market bias. Attention should be paid to any declared funding or the issue of a research grant, in order to check for a conflict of interest [ 2 ].

II. ABSTRACT: Reading the abstract is a quick way of getting to know the article and its purpose, major procedures and methods, main findings, and conclusions.

• Aim of the study: It should be well and clearly written.
• Materials and Methods: The study design and type of groups, type of randomization process, sample size, gender, age, and procedure rendered to each group and measuring tool(s) should be evidently mentioned.
• Results: The measured variables with their statistical analysis and significance.
• Conclusion: It must clearly answer the question of interest.

III. Introduction/Background section:

An excellent introduction will thoroughly include references to earlier work related to the area under discussion and express the importance and limitations of what is previously acknowledged [ 2 ].

-Why this study is considered necessary? What is the purpose of this study? Was the purpose identified before the study or a chance result revealed as part of ‘data searching?’

-What has been already achieved and how does this study be at variance?

-Does the scientific approach outline the advantages along with possible drawbacks associated with the intervention or observations?

IV. Methods and Materials section : Full details on how the study was actually carried out should be mentioned. Precise information is given on the study design, the population, the sample size and the interventions presented. All measurements approaches should be clearly stated [ 3 ].

V. Results section : This section should clearly reveal what actually occur to the subjects. The results might contain raw data and explain the statistical analysis. These can be shown in related tables, diagrams and graphs.

VI. Discussion section : This section should include an absolute comparison of what is already identified in the topic of interest and the clinical relevance of what has been newly established. A discussion on a possible related limitations and necessitation for further studies should also be indicated.

Does it summarize the main findings of the study and relate them to any deficiencies in the study design or problems in the conduct of the study? (This is called intention to treat analysis).

• Does it address any source of potential bias?
• Are interpretations consistent with the results?
• How are null findings interpreted?
• Does it mention how do the findings of this study relate to previous work in the area?
• Can they be generalized (external validity)?
• Does it mention their clinical implications/applicability?
• What are the results/outcomes/findings applicable to and will they affect a clinical practice?
• Does the conclusion answer the study question?
• -Is the conclusion convincing?
• -Does the paper indicate ethics approval?
• -Can you identify potential ethical issues?
• -Do the results apply to the population in which you are interested?
• -Will you use the results of the study?

Once you have answered the preliminary and key questions and identified the research method used, you can incorporate specific questions related to each method into your appraisal process or checklist.

1-What is the research question?

For a study to gain value, it should address a significant problem within the healthcare and provide new or meaningful results. Useful structure for assessing the problem addressed in the article is the Problem Intervention Comparison Outcome (PICO) method [ 3 ].

P = Patient or problem: Patient/Problem/Population:

It involves identifying if the research has a focused question. What is the chief complaint?

E.g.,: Disease status, previous ailments, current medications etc.,

I = Intervention: Appropriately and clearly stated management strategy e.g.,: new diagnostic test, treatment, adjunctive therapy etc.,

C= Comparison: A suitable control or alternative

E.g.,: specific and limited to one alternative choice.

O= Outcomes: The desired results or patient related consequences have to be identified. e.g.,: eliminating symptoms, improving function, esthetics etc.,

The clinical question determines which study designs are appropriate. There are five broad categories of clinical questions, as shown in [ Table/Fig-1 ].

[Table/Fig-1]:

Categories of clinical questions and the related study designs.

2- What is the study type (design)?

The study design of the research is fundamental to the usefulness of the study.

In a clinical paper the methodology employed to generate the results is fully explained. In general, all questions about the related clinical query, the study design, the subjects and the correlated measures to reduce bias and confounding should be adequately and thoroughly explored and answered.

Participants/Sample Population:

Researchers identify the target population they are interested in. A sample population is therefore taken and results from this sample are then generalized to the target population.

The sample should be representative of the target population from which it came. Knowing the baseline characteristics of the sample population is important because this allows researchers to see how closely the subjects match their own patients [ 4 ].

Sample size calculation (Power calculation): A trial should be large enough to have a high chance of detecting a worthwhile effect if it exists. Statisticians can work out before the trial begins how large the sample size should be in order to have a good chance of detecting a true difference between the intervention and control groups [ 5 ].

• Is the sample defined? Human, Animals (type); what population does it represent?
• Does it mention eligibility criteria with reasons?
• Does it mention where and how the sample were recruited, selected and assessed?
• Does it mention where was the study carried out?
• Is the sample size justified? Rightly calculated? Is it adequate to detect statistical and clinical significant results?
• Does it mention a suitable study design/type?
• Is the study type appropriate to the research question?
• Is the study adequately controlled? Does it mention type of randomization process? Does it mention the presence of control group or explain lack of it?
• Are the samples similar at baseline? Is sample attrition mentioned?
• All studies report the number of participants/specimens at the start of a study, together with details of how many of them completed the study and reasons for incomplete follow up if there is any.
• Does it mention who was blinded? Are the assessors and participants blind to the interventions received?
• Is it mentioned how was the data analysed?
• Are any measurements taken likely to be valid?

Researchers use measuring techniques and instruments that have been shown to be valid and reliable.

Validity refers to the extent to which a test measures what it is supposed to measure.

(the extent to which the value obtained represents the object of interest.)

• -Soundness, effectiveness of the measuring instrument;
• -What does the test measure?
• -Does it measure, what it is supposed to be measured?
• -How well, how accurately does it measure?

Reliability: In research, the term reliability means “repeatability” or “consistency”

Reliability refers to how consistent a test is on repeated measurements. It is important especially if assessments are made on different occasions and or by different examiners. Studies should state the method for assessing the reliability of any measurements taken and what the intra –examiner reliability was [ 6 ].

3-Selection issues:

The following questions should be raised:

• - How were subjects chosen or recruited? If not random, are they representative of the population?
• - Types of Blinding (Masking) Single, Double, Triple?
• - Is there a control group? How was it chosen?
• - How are patients followed up? Who are the dropouts? Why and how many are there?
• - Are the independent (predictor) and dependent (outcome) variables in the study clearly identified, defined, and measured?
• - Is there a statement about sample size issues or statistical power (especially important in negative studies)?
• - If a multicenter study, what quality assurance measures were employed to obtain consistency across sites?
• - Are there selection biases?
• • In a case-control study, if exercise habits to be compared:
• - Are the controls appropriate?
• - Were records of cases and controls reviewed blindly?
• - How were possible selection biases controlled (Prevalence bias, Admission Rate bias, Volunteer bias, Recall bias, Lead Time bias, Detection bias, etc.,)?
• • Cross Sectional Studies:
• - Was the sample selected in an appropriate manner (random, convenience, etc.,)?
• - Were efforts made to ensure a good response rate or to minimize the occurrence of missing data?
• - Were reliability (reproducibility) and validity reported?
• • In an intervention study, how were subjects recruited and assigned to groups?
• • In a cohort study, how many reached final follow-up?
• - Are the subject’s representatives of the population to which the findings are applied?
• - Is there evidence of volunteer bias? Was there adequate follow-up time?
• - What was the drop-out rate?
• - Any shortcoming in the methodology can lead to results that do not reflect the truth. If clinical practice is changed on the basis of these results, patients could be harmed.

Researchers employ a variety of techniques to make the methodology more robust, such as matching, restriction, randomization, and blinding [ 7 ].

Bias is the term used to describe an error at any stage of the study that was not due to chance. Bias leads to results in which there are a systematic deviation from the truth. As bias cannot be measured, researchers need to rely on good research design to minimize bias [ 8 ]. To minimize any bias within a study the sample population should be representative of the population. It is also imperative to consider the sample size in the study and identify if the study is adequately powered to produce statistically significant results, i.e., p-values quoted are <0.05 [ 9 ].

4-What are the outcome factors and how are they measured?

• -Are all relevant outcomes assessed?
• -Is measurement error an important source of bias?

5-What are the study factors and how are they measured?

• -Are all the relevant study factors included in the study?
• -Have the factors been measured using appropriate tools?

Data Analysis and Results:

- Were the tests appropriate for the data?

- Are confidence intervals or p-values given?

• How strong is the association between intervention and outcome?
• How precise is the estimate of the risk?
• Does it clearly mention the main finding(s) and does the data support them?
• Does it mention the clinical significance of the result?
• Is adverse event or lack of it mentioned?
• Are all relevant outcomes assessed?
• Was the sample size adequate to detect a clinically/socially significant result?
• Are the results presented in a way to help in health policy decisions?
• Is there measurement error?
• Is measurement error an important source of bias?

Confounding Factors:

A confounder has a triangular relationship with both the exposure and the outcome. However, it is not on the causal pathway. It makes it appear as if there is a direct relationship between the exposure and the outcome or it might even mask an association that would otherwise have been present [ 9 ].

6- What important potential confounders are considered?

• -Are potential confounders examined and controlled for?
• -Is confounding an important source of bias?

7- What is the statistical method in the study?

• -Are the statistical methods described appropriate to compare participants for primary and secondary outcomes?
• -Are statistical methods specified insufficient detail (If I had access to the raw data, could I reproduce the analysis)?
• -Were the tests appropriate for the data?
• -Are confidence intervals or p-values given?
• -Are results presented as absolute risk reduction as well as relative risk reduction?

Interpretation of p-value:

The p-value refers to the probability that any particular outcome would have arisen by chance. A p-value of less than 1 in 20 (p<0.05) is statistically significant.

• When p-value is less than significance level, which is usually 0.05, we often reject the null hypothesis and the result is considered to be statistically significant. Conversely, when p-value is greater than 0.05, we conclude that the result is not statistically significant and the null hypothesis is accepted.

Confidence interval:

Multiple repetition of the same trial would not yield the exact same results every time. However, on average the results would be within a certain range. A 95% confidence interval means that there is a 95% chance that the true size of effect will lie within this range.

8- Statistical results:

• -Do statistical tests answer the research question?

Are statistical tests performed and comparisons made (data searching)?

Correct statistical analysis of results is crucial to the reliability of the conclusions drawn from the research paper. Depending on the study design and sample selection method employed, observational or inferential statistical analysis may be carried out on the results of the study.

It is important to identify if this is appropriate for the study [ 9 ].

• -Was the sample size adequate to detect a clinically/socially significant result?
• -Are the results presented in a way to help in health policy decisions?

Clinical significance:

Statistical significance as shown by p-value is not the same as clinical significance. Statistical significance judges whether treatment effects are explicable as chance findings, whereas clinical significance assesses whether treatment effects are worthwhile in real life. Small improvements that are statistically significant might not result in any meaningful improvement clinically. The following questions should always be on mind:

• -If the results are statistically significant, do they also have clinical significance?
• -If the results are not statistically significant, was the sample size sufficiently large to detect a meaningful difference or effect?

9- What conclusions did the authors reach about the study question?

Conclusions should ensure that recommendations stated are suitable for the results attained within the capacity of the study. The authors should also concentrate on the limitations in the study and their effects on the outcomes and the proposed suggestions for future studies [ 10 ].

• -Are the questions posed in the study adequately addressed?
• -Are the conclusions justified by the data?
• -Do the authors extrapolate beyond the data?
• -Are shortcomings of the study addressed and constructive suggestions given for future research?
• -Bibliography/References:

Do the citations follow one of the Council of Biological Editors’ (CBE) standard formats?

10- Are ethical issues considered?

If a study involves human subjects, human tissues, or animals, was approval from appropriate institutional or governmental entities obtained? [ 10 , 11 ].

Critical appraisal of RCTs: Factors to look for:

• Allocation (randomization, stratification, confounders).
• Follow up of participants (intention to treat).
• Data collection (bias).
• Sample size (power calculation).
• Presentation of results (clear, precise).
• Applicability to local population.

[ Table/Fig-2 ] summarizes the guidelines for Consolidated Standards of Reporting Trials CONSORT [ 12 ].

[Table/Fig-2]:

Summary of the CONSORT guidelines.

Critical appraisal of systematic reviews: provide an overview of all primary studies on a topic and try to obtain an overall picture of the results.

In a systematic review, all the primary studies identified are critically appraised and only the best ones are selected. A meta-analysis (i.e., a statistical analysis) of the results from selected studies may be included. Factors to look for:

• Literature search (did it include published and unpublished materials as well as non-English language studies? Was personal contact with experts sought?).
• Quality-control of studies included (type of study; scoring system used to rate studies; analysis performed by at least two experts).
• Homogeneity of studies.

[ Table/Fig-3 ] summarizes the guidelines for Preferred Reporting Items for Systematic reviews and Meta-Analyses PRISMA [ 13 ].

[Table/Fig-3]:

Summary of PRISMA guidelines.

Critical appraisal is a fundamental skill in modern practice for assessing the value of clinical researches and providing an indication of their relevance to the profession. It is a skills-set developed throughout a professional career that facilitates this and, through integration with clinical experience and patient preference, permits the practice of evidence based medicine and dentistry. By following a systematic approach, such evidence can be considered and applied to clinical practice.

Financial or other Competing Interests

• Skip to main content
• Keyboard shortcuts for audio player

NPR fact-checked the Harris-Trump presidential debate. Here's what we found

Vice President and Democratic presidential candidate Kamala Harris and former President and Republican presidential candidate Donald Trump speak during a presidential debate. Saul Loeb/AFP via Getty Images hide caption

Vice President Harris and former President Donald Trump faced off Tuesday in their first — and possibly only — debate of the 2024 campaign, taking questions on key issues like the border, the economy and abortion.

With the candidates virtually tied in the polls, and just 55 days until Election Day, Trump and Harris sought to define their visions for America in front of a national audience and deflect attacks from the other side.

NPR reporters fact-checked the candidates' claims in real time . Here's what they found:

TRUMP: "I had no inflation, virtually no inflation. They had the highest inflation, perhaps in the history of our country, because I've never seen a worse period of time. People can't go out and buy cereal or bacon or eggs or anything else."

Inflation soared to a four-decade high of 9.1% in 2022, according to the consumer price index. While inflation has since fallen to 2.9% (as of July), prices — particularly food prices — are still higher than many Americans would like.

Other countries have also faced high inflation in the wake of the pandemic, as tangled supply chains struggled to keep pace with surging demand. Russia’s invasion of Ukraine also fueled inflation by driving up energy and food prices worldwide.

Government spending in the U.S. under both the Biden-Harris administration and Trump also may have contributed, putting more money in people’s pockets and enabling them to keep spending in the face of high prices.

While high prices are a source of frustration for many Americans, the average worker has more buying power today than she did before the pandemic. Since February 2020 (just before the pandemic took hold in the U.S.), consumer prices have risen 21.6% while average wages have risen 23%.

Many prices were depressed early in the pandemic, however, so the comparison is less flattering if you start the clock when President Biden and Vice President Harris took office. Since early 2021, consumer prices have risen 19.6%, while average wages have risen 16.9%. Wage gains have been outpacing price increases for over a year, so that gap should eventually close.

— NPR economics correspondent Scott Horsley

2024 Election

Taylor swift endorses kamala harris in instagram post after the debate.

HARRIS: "Donald Trump left us the worst unemployment since the Great Depression."

At the height of the Great Depression in 1933, the national unemployment rate was near 25%, according to the Franklin D. Roosevelt Presidential Library.

At the start of the COVID pandemic, the unemployment rate peaked at 14.8% in April 2020, a level not seen since 1948, according to the Congressional Research Service.

But by the time Trump left office, unemployment had fallen to a lower, but still elevated, level. The January 2021 unemployment rate was 6.3%.

— NPR producer Lexie Schapitl

Immigration

TRUMP: "You see what's happening with towns throughout the United States. You look at Springfield, Ohio, you look at Aurora in Colorado. They are taking over the towns. They're taking over buildings. They're going in violently. These are the people that she and Biden let into our country, and they're destroying our country. They're dangerous. They're at the highest level of criminality, and we have to get them out."

Trump attacked Harris and Biden's records on immigration, arguing that they're failing to stem people from other countries from entering the U.S. and causing violence.

In the last two years, more than 40,000 Venezuelan immigrants have arrived in the Denver metro area. And it is true that many now live in Aurora.

A few weeks ago, a video of gang members in an Aurora, Colo., apartment building had right-wing media declaring the city's takeover by Venezuelan gangs. NPR looked into these claims .

Indian migrants drive surge in northern U.S. border crossings

Shortly after the video appeared, Colorado's Republican Party sent a fundraising letter claiming the state is under violent attack, and Venezuelan gangs have taken over Aurora.

It's also true Aurora police have recently arrested 10 members of a Venezuelan gang called Tren de Aragua. But Aurora's interim police chief, Heather Morris, says there's no evidence of a gang takeover of apartment buildings in her city.

What's more, violent crime — including murder, robbery and rape — is way down nationwide, according to the most recent data from the FBI . Notably, analysts predict violent crime rates this year will fall back down to where they were before they surged during the pandemic and may even approach a 50-year low.

Trump also claims that migrants are driving up crime rates in the U.S. That is not true. Researchers from Stanford University found that since the 1960s, immigrants have been 60% less likely to be incarcerated than people born in the U.S. The Cato Institute, a libertarian think tank, found undocumented immigrants in Texas were 37% less likely to be convicted of a crime.

— NPR immigration correspondent Jasmine Garsd and criminal justice reporter Meg Anderson

TRUMP: "In Springfield, they're eating the dogs. The people that came in, they're eating the cats. They're eating the pets of the people that live there."

This remark refers to a debunked, dehumanizing claim that Haitian migrants living in Springfield, Ohio, are abducting pets and eating them .

Untangling Disinformation

Jd vance spreads debunked claims about haitian immigrants eating pets.

The claim, which local police say is baseless, first circulated among far-right activists, local Republicans and neo-Nazis before being picked up by congressional leaders, vice presidential candidate JD Vance and others. A well-known advocate for the Haitian community says she received a wave of racist harassment after Vance shared the theory on social media.

The Springfield News-Sun reported that local police said that incidents of pets being stolen or eaten were "not something that's on our radar right now." The paper said the unsubstantiated claim seems to have started with a post in a Springfield Facebook group that was widely shared across social media.

The claim is the latest example of Trump leaning into anti-immigrant rhetoric. Since entering the political arena in 2015, Trump accused immigrants of being criminals, rapists, or "poisoning the blood of our nation."

— NPR immigration correspondent Jasmine Garsd

TRUMP: "A lot of these illegal immigrants coming in, [Democrats] are trying to get them to vote."

It is illegal for noncitizens to vote in federal elections, and there is no credible evidence that it has happened in significant numbers, or that there is an effort underway to illegally register undocumented immigrants to vote this election.

Voter registration forms require voters to sign an oath — under penalty of perjury — that they are U.S. citizens. If a noncitizen lies about their citizenship on a registration form and votes, they have created a paper trail of a crime that is punishable with jail time and deportation.

“The deterrent is incredibly strong,” David Becker, executive director of the Center for Election Innovation and Research, told NPR.

Illegal crossings hit Biden-era low as migrants wait longer for entry

Election officials routinely verify information on voter registration forms, which ask registrants for either a driver’s license number or the last four digits of Social Security numbers.

In 2016, the Brennan Center for Justice surveyed local election officials in 42 jurisdictions with high immigrant populations and found 30 cases of suspected noncitizens voting out of 23.5 million votes cast, or 0.0001%.

Georgia Secretary of State Brad Raffensperger launched an audit in 2022 that found fewer than 1,700 suspected noncitizens had attempted to register to vote over the past 25 years. None were able to vote.

— NPR disinformation reporter Jude Joffe-Block

TRUMP: "[Harris] was the border czar. Remember that she was the border czar."

Republicans have taken to calling Harris the "border czar" as a way to blame her for increased migration to the U.S. and what they see as border security policy failures of the Biden administration.

There is no actual "border czar" position. In 2021, President Biden tasked Harris with addressing the root causes of migration from Central America.

As Republicans attack Harris on immigration, here’s what her California record reveals

The "root causes strategy ... identifies, prioritizes, and coordinates actions to improve security, governance, human rights, and economic conditions in the region," the White House said in a statement. "It integrates various U.S. government tools, including diplomacy, foreign assistance, public diplomacy, and sanctions."

While Harris has been scrutinized on the right, immigration advocates have also criticized Harris, including for comments in 2021 where she warned prospective migrants, "Do not come."

TRUMP: "You could do abortions in the seventh month, the eighth month, the ninth month, and probably after birth."

As ABC News anchor Linsey Davis mentioned during her real-time fact check, there is no state where it is legal to kill a baby after birth (Trump called it "execution"). A report from KFF earlier this year also noted that abortions “after birth” are illegal in every state.

According to the Pew Research Center, the overwhelming majority of abortions — 93% — take place during the first trimester. Pew says 1% take place after 21 weeks. Most of those take place before 24 weeks, the approximate timeline for fetal viability, according to a report by KFF Health News.

Trump repeats the false claim that Democrats support abortion 'after birth' in debate

A separate analysis from KFF earlier this year noted that later abortions are expensive to obtain and offered by relatively few providers, and often occur because of medical complications or because patients face barriers earlier in their pregnancies.

“Nowhere in America is a woman carrying a pregnancy to term and asking for an abortion. That isn’t happening; it’s insulting to the women of America,” Harris said.

Harris also invoked religion in her response, arguing that “one does not have to abandon their faith” to agree that the government should not control reproductive health decisions.

As Davis also noted, Trump has offered mixed messages about abortion over the course of the campaign. He has bragged about his instrumental role in overturning Roe v. Wade , while appearing to backpedal on an issue that polling makes clear is a liability for Republicans.

— NPR political correspondent Sarah McCammon

Afghanistan

TRUMP: The U.S. withdrawal from Afghanistan "was one of the most incompetently handled situations anybody has ever seen."

Trump and Republicans in Congress say President Biden is to blame for the fall of Kabul to the Taliban three years ago, and the chaotic rush at the airport where 13 U.S. troops died in a suicide bomb attack that killed nearly 200 Afghan civilians trying to flee. Of late, Republicans have been emphasizing Harris’ role . But the Afghanistan war spanned four U.S. presidencies , and it's important to note that it was the Trump administration that signed a peace deal that was basically a quick exit plan.

Trump regularly claims there were no casualties in Afghanistan for 18 months under his administration, and it’s not true, according to Pentagon records.

— NPR veterans correspondent Quil Lawrence

Military policy

HARRIS: “There is not one member of the military who is in active duty in a combat zone in any war zone around the world for the first time this century.”

This is a common administration talking point, and it's technically true. But thousands of troops in Iraq and on the Syrian border are still in very dangerous terrain. U.S. troops died in Jordan in January on a base that keeps watch over the war with ISIS in Syria.

HARRIS: "I will not ban fracking. I have not banned fracking as vice president United States, and in fact, I was the tie-breaking vote on the inflation Reduction Act which opened new leases for fracking."

When she first ran for president in 2019, Harris had said she was firmly in favor of banning fracking — a stance she later abandoned when she joined President Biden’s campaign as his running mate.

In an interview with CNN last month, Harris attempted to explain why her position has changed from being against fracking to being in favor of it.

“What I have seen is that we can grow, and we can increase a clean energy economy without banning fracking,” Harris told CNN’s Dana Bash.

Harris says she won't ban fracking. What to know about the controversial topic

Under the Biden-Harris administration, the U.S. produced a record amount of oil last year — averaging 12.9 million barrels per day. That eclipsed the previous record of 12.3 million barrels per day, set under Trump in 2019. 2023 was also a record year for domestic production of natural gas . Much of the domestic boom in oil and gas production is the result of hydraulic fracturing or “fracking” techniques .

In addition to record oil and gas production, the Biden-Harris administration has also coincided with rapid growth of solar and wind power . Meanwhile, coal has declined as a source of electricity.

Health care

TRUMP: "I had a choice to make: Do I save [the Affordable Care Act] and make it as good as it can be, or do I let it rot? And I saved it."

During his presidency, Trump undermined the Affordable Care Act in many ways — for instance, by slashing funding for advertising and free "navigators" who help people sign up for a health insurance plan on HealthCare.gov. And rather than deciding to "save" the ACA, he tried hard to get Congress to repeal it, and failed. When pushed Tuesday on what health policy he would put in its place, he said he has "concepts of a plan."

Shots - Health News

Amid medicaid's 'unwinding,' many states work to expand health care access.

The Biden administration has reversed course from Trump's management of the Affordable Care Act. Increased subsidies have made premiums more affordable in the marketplaces, and enrollment has surged. The uninsurance rate has dropped to its lowest point ever during the Biden administration.

The Affordable Care Act was passed in 2010 and is entrenched in the health care system. Republicans successfully ran against Obamacare for about a decade, but it has faded as a campaign issue this year.

— NPR health policy correspondent Selena Simmons-Duffin

• DOI: 10.36849/JDD.8221
• Corpus ID: 270256130

Heatmap Evaluation of Facial Hydration Using a Novel Python Program.

• Thu Q Nguyen , Christine Emesiani , M. Meckfessel
• Published in Journal of Drugs in… 1 May 2024
• Medicine, Computer Science

Related Papers

Showing 1 through 3 of 0 Related Papers

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Research on the optimization method of visual sensor calibration combining convex lens imaging with the bionic algorithm of wolf pack predation.

1. Introduction

2. nonlinear imaging model, 3. principle of the bionic algorithm of wolf pack predation (wpp), 4.1. the law of imaging by a convex lens, 4.2. reverse learning strategies for convex lens imaging techniques, 4.3. camera calibration using cli-wpp, 5. experiment and analysis, 6. result analysis and discussion.

7. Conclusions

Author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• Zhu, D.; Cheng, W.; Zhang, Y.; Liu, H. Multi-camera joint calibration algorithm for guiding machining robot positioning considering ambient light and error uncertainty. Opt. Lasers Eng. 2024 , 178 , 108251. [ Google Scholar ] [ CrossRef ]
• Liu, J.; He, P.; Hou, X.; Xiao, J.; Wen, L.; Lv, M. A Target-free Vision-based Method for Out-of-plane Vibration Measurement Using Projection Speckle and Camera Self-calibration Technology. Eng. Struct. 2024 , 303 , 117416. [ Google Scholar ]
• Ge, J.; Deng, Z.; Li, Z.; Liu, T.; Zhuo, R.; Chen, X. Adaptive parameter optimization approach for robotic grinding of weld seam based on laser vision sensor. Robot. Comput.-Integr. Manuf. 2023 , 82 , 102540. [ Google Scholar ] [ CrossRef ]
• Che, M.; Duan, Z.; Lan, Z.; Yi, S. Scene Reconstruction Algorithm for Unstructured Weak-Texture Regions Based on Stereo Vision. Appl. Sci. 2023 , 13 , 6407. [ Google Scholar ] [ CrossRef ]
• Li, S.; Yoon, H.S. Enhancing Camera Calibration for Traffic Surveillance with an Integrated Approach of Genetic Algorithm and Particle Swarm Optimization. Sensors 2024 , 24 , 1456. [ Google Scholar ] [ CrossRef ]
• Fu, W.; Wu, L. Camera Calibration Method Based on Improved Differential Evolution Particle Swarm Optimization. Meas. Control. Technol. 2023 , 56 , 27–33. [ Google Scholar ] [ CrossRef ]
• Lu, X.; Tang, X. Visual Localization of Workpieces Based on Evolutionary Algorithm and Its Application in Industrial Robot. In Proceedings of the 2023 3rd International Conference on Intelligent Technologies (CONIT), Hubli, India, 23–25 June 2023; pp. 1–5. [ Google Scholar ]
• Ajjah, H.A.; Alboabidallah, A.H.; Mohammed, M.U.; Hussain, A.A. Modelling and Estimation of Interior Orientation of Non-Metric Cameras using Artificial Intelligence. J. Tech. 2023 , 5 , 39–45. [ Google Scholar ] [ CrossRef ]
• Xu, C.; Liu, Y.; Xiao, Y.; Cao, J. Camera Intrinsics Optimization Method Based on Improved Particle Swarm Optimization Algorithm. Laser Optoelectron. Prog. 2020 , 57 , 041514. [ Google Scholar ]
• To.Sriwong, K.; Paripurana, S.; Vanichchanunt, P. Study of Particle Swarm Optimization Parameter Tuning for Camera Calibration. In Proceedings of the 2023 International Technical Conference on Circuits/Systems, Computers, and Communications (ITC-CSCC), Jeju, Republic of Korea, 25–28 June 2023; pp. 1–6. [ Google Scholar ]
• Shang, H.; Ni, S.; Su, Z. An Optimization Method for Visual Sensor Calibration Based on an Improved Sparrow Search Algorithm. Comput. Technol. Dev. 2023 , 3 , 146–151, 160. [ Google Scholar ]
• Daviran, M.; Ghezelbash, R.; Maghsoudi, A. WPPKM: A novel hybrid optimization algorithm for geochemical anomaly detection based on grey wolf optimizer and K-means clustering. Geochemistry 2024 , 84 , 126036. [ Google Scholar ] [ CrossRef ]
• Zhu, H. Research on Intelligent Driving Vehicle Recognition and Distance Measurement Based on Machine Vision. Electron. Prod. 2021 , 8 , 49–50, 75. [ Google Scholar ]
• Greer, R.; Gopalakrishnan, A.; Keskar, M.; Trivedi, M. Patterns of vehicle lights: Addressing complexities of camera-based vehicle light datasets and metrics. Pattern Recognit. Lett. 2024 , 178 , 209–215. [ Google Scholar ] [ CrossRef ]
• Wang, Y.; Qin, X.; Jia, W.; Lei, J.; Wang, D.; Feng, T.; Zeng, Y.; Song, J. Mult objective Energy Consumption Optimization of a Flying–Walking Power Transmission Line Inspection Robot during Flight Missions Using Improved NSGA-II. Appl. Sci. 2024 , 14 , 1637. [ Google Scholar ] [ CrossRef ]
• Khishe, M.; Azar, O.P.; Hashemzadeh, E. Variable-length CNNs evolved by digitized chimp optimization algorithm for deep learning applications. Multimed. Tools Appl. 2024 , 83 , 2589–2607. [ Google Scholar ] [ CrossRef ]
• Yu, F.; Guan, J.; Wu, H.; Chen, Y.; Xia, X. Lens imaging opposition-based learning for differential evolution with Cauchy perturbation. Appl. Soft Comput. 2024 , 152 , 111211. [ Google Scholar ] [ CrossRef ]
• Ren, J.; Cao, Z. Camera Calibration Optimization Method Based on Improved Wind-Driven Optimization Algorithm. Comput. Eng. Des. 2021 , 42 , 942–948. [ Google Scholar ]

Click here to enlarge figure

Share and Cite

Wu, Q.; Miao, J.; Liu, Z.; Chang, J. Research on the Optimization Method of Visual Sensor Calibration Combining Convex Lens Imaging with the Bionic Algorithm of Wolf Pack Predation. Sensors 2024 , 24 , 5926. https://doi.org/10.3390/s24185926

Wu Q, Miao J, Liu Z, Chang J. Research on the Optimization Method of Visual Sensor Calibration Combining Convex Lens Imaging with the Bionic Algorithm of Wolf Pack Predation. Sensors . 2024; 24(18):5926. https://doi.org/10.3390/s24185926

Wu, Qingdong, Jijun Miao, Zhaohui Liu, and Jiaxiu Chang. 2024. "Research on the Optimization Method of Visual Sensor Calibration Combining Convex Lens Imaging with the Bionic Algorithm of Wolf Pack Predation" Sensors 24, no. 18: 5926. https://doi.org/10.3390/s24185926

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals