Research Impact Assessment

Kostoff, Ronald N.; Averch, Harvey A.; Chubin, Daryl E.

doi:10.1177/0193841x9401800101

Cited by 28 publications

(37 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the paper is targeted toward research program peer review, most of the principles are applicable to multiple types of peer review. The author's experience, based on examining the peer review literature, conducting many peer review experiments (e.g., Kostoff, 1988), and managing hundreds of peer reviews, leads to the following conclusions about the factors critical to high-quality peer review (Kostoff, 1995(Kostoff, , 1997a(Kostoff, , 2001b, whether it be of proposals, programs, procedures, manuscripts, faculty performance, or research dissertations.…”

Section: Appendix -Executive Summary/peer-review Principles (Modified)mentioning

confidence: 99%

“…For evaluating basic research proposals, the three main criteria are research merit, research approach, and team quality (DOE, 1982;Kostoff, 1992Kostoff, , 1997a. For research sponsored by a mission-oriented organization, a fourth criterion related to mission relevance is useful.…”

Section: ) Selection Of Evaluation Criteriamentioning

confidence: 99%

“…For S&T areas that have some similarity, use of common experts (on the evaluation teams) with broad backgrounds that overlap the disciplines can provide some degree of standardization (Kostoff, 1988(Kostoff, , 1997a. For very disparate S&T areas, some allowances need to be made for the relative strategic value of each discipline to the organization, and arbitrary corrections applied for benefit estimation differences and biases.…”

Section: ) Normalization Across Technical Disciplinesmentioning

confidence: 99%

“…The next critical factor for quality S&T evaluations is cost (ASTEC, 1991;Buechner, 1974;Hensley, 1980;Kostoff, 1995Kostoff, , 1997a. The true total costs of peer review can be considerable, but tend to be ignored or understated in most reported cases.…”

Section: ) Cost Of Sandt Evaluationsmentioning

confidence: 99%

“…The true total costs of peer review can be considerable, but tend to be ignored or understated in most reported cases. For high quality peer reviews, where sufficient expertise is represented on the review group, total real costs will dominate direct costs (Kostoff, 1995(Kostoff, , 1997a. The major contributor to total costs is the time of all the individuals involved in executing the review, including staff, reviewer, and presenter time.…”

Section: ) Cost Of Sandt Evaluationsmentioning

confidence: 99%

See 4 more Smart Citations

Science and Technology Peer Review: GPRA

Kostoff¹

2003

Self Cite

View full text Add to dashboard Cite

Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.

show abstract

Section: Appendix -Executive Summary/peer-review Principles (Modified)mentioning

confidence: 99%

Section: ) Selection Of Evaluation Criteriamentioning

confidence: 99%

Section: ) Normalization Across Technical Disciplinesmentioning

confidence: 99%

Section: ) Cost Of Sandt Evaluationsmentioning

confidence: 99%

Section: ) Cost Of Sandt Evaluationsmentioning

confidence: 99%

See 3 more Smart Citations

Science and Technology Peer Review: GPRA

Kostoff¹

2003

Self Cite

View full text Add to dashboard Cite

show abstract

Federal research impact assessment: State-of-the-art

Kostoff

1994

J. Am. Soc. Inf. Sci.

Self Cite

View full text Add to dashboard Cite

This article describes the practice of federal research impact assessment. Evaluation of research impact is described for three cases: research selection, where the work has not yet been performed; research review, where work and results are ongoing; and ex‐post research assessment, where research has been completed and results can be tracked. Retrospective methods (such as projects Hindsight and TRACES), qualitative methods (such as peer review), and quantitative methods (such as cost‐benefit analysis and bibliometrics) are described. While peer review in its broadest sense is the most widely used method in research selection, review, and ex‐post assessment, it has its deficiencies, and there is no single method which provides a complete impact evaluation. © 1994 John Wiley & Sons, Inc.

show abstract

Citation mining: Integrating text mining and bibliometrics for research user profiling

Kostoff

Río

Humenik³

et al. 2001

J. Am. Soc. Inf. Sci.

Self Cite

102

View full text Add to dashboard Cite

Identifying the users and impact of research is important for research performers, managers, evaluators, and sponsors. It is important to know whether the audience reached is the audience desired. It is useful to understand the technical characteristics of the other research/development/applications impacted by the originating research, and to understand other characteristics (names, organizations, countries) of the users impacted by the research. Because of the many indirect pathways through which fundamental research can impact applications, identifying the user audience and the research impacts can be very complex and time consuming. The purpose of this article is to describe a novel approach for identifying the pathways through which research can impact other research, technology development, and applications, and to identify the technical and infrastructure characteristics of the user population. A novel literature-based approach was developed to identify the user community and its characteristics. The research performed is characterized by one or more articles accessed by the Science Citation Index (SCI) database, beccause the SCI's citation-based structure enables the capability to perform citation studies easily.The user community is characterized by the articles in the SCI that cite the original research articles, and that cite the succeeding generations of these articles as well. Text mining is performed on the citing articles to identify the technical areas impacted by the research, the relationships among these technical areas, and relationships among the technical areas and the infrastructure (authors, journals, organizations). A key component of text mining, concept clustering, was used to provide both a taxonomy of the citing articles' technical themes and further technical insights based on theme relationships arising from the grouping process. Bibliometrics is performed on the citing articles to profile the user characteristics. Citation Mining, this integration of citation bibliometrics and text mining, is applied to the 307 first generation citing articles of a fundamental physics article on the dynamics of vibrating sand-piles. Most of the 307 citing articles were basic research whose main themes were aligned with those of the cited article. However, about 20% of the citing articles were research or development in other disciplines, or development within the same discipline. The text mining alone identified the intradiscipline applications and extradiscipline impacts and applications; this was confirmed by detailed reading of the 307 abstracts. The combination of citation bibliometrics and text mining provides a synergy unavailable with each approach taken independently. Furthermore, text mining is a REQUIREMENT for a feasible comprehensive research impact determination. The integrated multigeneration citation analysis required for broad research impact determination of highly cited articles will produce thousands or tens or hundreds of thousands of citing article Abstracts. Text mining allows...

show abstract

Research Impact Assessment

Cited by 28 publications

References 0 publications

Science and Technology Peer Review: GPRA

Science and Technology Peer Review: GPRA

Federal research impact assessment: State-of-the-art

Citation mining: Integrating text mining and bibliometrics for research user profiling

Contact Info

Product

Resources

About