The performance based research fund in New Zealand: taking stock and looking forward

Buckle, Robert A.; Creedy, John

doi:10.1080/00779954.2022.2156382

New Zealand Economic Papers

2022

DOI: 10.1080/00779954.2022.2156382

|View full text |Cite

The performance based research fund in New Zealand: taking stock and looking forward

Robert A. Buckle

John Creedy

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2022

2024

Publication Types

Select...

Article2

Relationship

Self Cite1

Independent1

Authors

Journals

Cited by 2 publications

(1 citation statement)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The main response to this switch in funding has been growth in research quality as per the QE process, and growth in external research income. There has been very little growth in research degree completions; see Buckle and Creedy (2022, Appendix A. )…”

mentioning

confidence: 99%

Methods to evaluate institutional responses to performance‐based research funding systems

Buckle

Creedy

2022

Australian Economic Papers

Self Cite

View full text Add to dashboard Cite

Performance‐based research funding systems (PBRFS) have been introduced in many countries for allocating funding to research institutions. There continues to be considerable debate about the effectiveness and consequences of these systems. This paper suggests several methods that can be applied to evaluate how institutions respond to new incentives created by the introduction of a PBRFS. The methods are illustrated using longitudinal data from the New Zealand PBRFS, which assesses institutional performance and allocates funds based on individual researcher performance, although not all methods require individual data. Longitudinal data enable the identification of entry, exit and quality transformation of researchers and the contribution of these dynamics to changes in university and discipline research quality. The approach enables a deeper understanding of individual and institutional responses to PBRFSs, the sustainability of changes and the contributions of changes in researcher quality and discipline composition to changes in institutional performance.

show abstract

mentioning

confidence: 99%

Methods to evaluate institutional responses to performance‐based research funding systems

Buckle

Creedy

2022

Australian Economic Papers

Self Cite

View full text Add to dashboard Cite

show abstract

Evaluating research quality with Large Language Models: An analysis of ChatGPT’s effectiveness with different settings and inputs

Thelwall

2024

Journal of Data and Information Science

View full text Add to dashboard Cite

Purpose Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process. Design/methodology/approach This article assesses which ChatGPT inputs (full text without tables, figures, and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts. Findings The optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66). Research limitations The data is a convenience sample of the work of a single author, it only includes one field, and the scores are self-evaluations. Practical implications The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones. Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing. Originality/value This is the first systematic comparison of the impact of different prompts, parameters and inputs for ChatGPT research quality evaluations.

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

The performance based research fund in New Zealand: taking stock and looking forward

Cited by 2 publications

References 35 publications

Methods to evaluate institutional responses to performance‐based research funding systems

Methods to evaluate institutional responses to performance‐based research funding systems

Evaluating research quality with Large Language Models: An analysis of ChatGPT’s effectiveness with different settings and inputs

Contact Info

Product

Resources

About