Assessing critical thinking in higher education: the HEIghten™ approach and preliminary validity evidence

Liu, Ou Lydia; Mao, Liyang; Frankel, Lois; Xu, Jia

doi:10.1080/02602938.2016.1168358

Cited by 76 publications

(53 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Results showed that as high school and cumulative college GPA increased, quantitative literacy scores also increased, suggesting that pre-college and within-college academic performance has a relationship with students' quantitative literacy performance. These results are consistent with previous research also investigating the relationship between SLO performance and GPA (e.g., Kuncel et al 2001;ACT 2012;Liu and Roohr 2013;Liu et al 2016;Graduate Management Admissions Council 2017).…”

Section: Positive Relationships With Gpa and College Admissions Scoressupporting

confidence: 82%

“…Students in the second highest high school GPA category (3.00 to 3.49) also scored statistically significantly higher than the two bottom categories with moderate effect sizes ranging from 0.38 to 0.48. In terms of the magnitude of these differences, results were comparable to Liu et al (2016). Similar results were found for college GPA.…”

Section: Relationships With Other Variablessupporting

confidence: 76%

“…For instance, Shavelson (2010) found correlations ranging from 0.55 to 0.57 between scores on the CLA and SAT. More recently, Liu et al (2016) found a correlation of 0.54 between HEIghten Critical Thinking score and an SAT/ACT composite score. Those authors noted that the disattenuated correlations ranged from 0.63 to 0.71.…”

Section: Number Of College Mathematics Classesmentioning

confidence: 99%

“…Note, however, that even a small effect can be viewed as important depending on the theory being tested (Gall et al 2007). To give these effects context, we will consider results from other studies such as Liu et al (2016) who conducted ANOVAs to evaluate performance differences on the HEIghten Critical Thinking measure.…”

Section: Test Reliabilitymentioning

confidence: 99%

See 3 more Smart Citations

Preliminary Evaluation of the Psychometric Quality of HEIghtenTM Quantitative Literacy

Roohr¹,

Lee²,

Xu³

et al. 2017

Numeracy

Self Cite

View full text Add to dashboard Cite

Quantitative literacy has been identified as an important student learning outcome (SLO) by both the higher education and workforce communities. This paper aims to provide preliminary evidence of the psychometric quality of the pilot forms for HEIghten quantitative literacy, a next-generation SLO assessment for students in higher education. We evaluated the psychometric quality of the test items (e.g., item analyses), individual-and group-level reliability, the relationship with student performance and related variables (e.g., grade point average) as well as student perceptions, and differences across college-related and demographic subgroups. Our study used data from a pilot test administered to over 1,500 students at 23 higher education institutions in the United States. Results showed that (a) overall, items were functioning well, but a small portion of items should be dropped due to unsatisfactory performance; (b) correlations across sub-areas of the assessment were very high indicating that the assessment may be unidimensional; (c) reliability estimates similar to existing SLO assessments were found at both individual and group levels; (d) assessment scores correlated positively with high school and college GPA, number of math college courses, self-rated quantitative literacy skills, and college admissions scores; (e) students had positive perceptions about the assessment; and (f) performance differences were found across institution type, college majors, gender, racial/ethnic groups, and language groups, but not across credit-hour categories. Implications for operational test development and understanding of quantitative literacy performance are discussed. Zhen Wang is a Senior Psychometrician at the Educational Testing Service (ETS). She holds a Ph.D. in Educational Measurement and Research Methodology from the University of British Columbia. Currently, she is the statistical coordinator for ETS' HEIghten® Outcomes Assessment Suite, Major Field Test, Success Navigator, Proficiency Profile, and iSkills projects. She also works on the research projects related to automated scoring and higher education. Her research interests include item response theory, structural equation models, hierarchical linear models, rater models, and equating and scaling techniques.

show abstract

Section: Positive Relationships With Gpa and College Admissions Scoressupporting

confidence: 82%

Section: Relationships With Other Variablessupporting

confidence: 76%

Section: Number Of College Mathematics Classesmentioning

confidence: 99%

Section: Test Reliabilitymentioning

confidence: 99%

See 2 more Smart Citations

Preliminary Evaluation of the Psychometric Quality of HEIghtenTM Quantitative Literacy

Roohr¹,

Lee²,

Xu³

et al. 2017

Numeracy

Self Cite

View full text Add to dashboard Cite

show abstract

“…The pilot study includes 5 forms (versions) for the critical thinking test. The five forms were developed under the same test specification and their scores were linked to each other and can be used interchangeably (Liu, et al, 2016).…”

Section: The Studymentioning

confidence: 99%

Exploring Relationships Between Writing and Broader Outcomes With Automated Writing Evaluation

Burstein

McCaffrey

Klebanov

et al. 2017

Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

No significant body of research examines writing achievement and the specific skills and knowledge in the writing domain for postsecondary (college) students in the U.S., even though many at-risk students lack the prerequisite writing skills required to persist in their education. This paper addresses this gap through a novel exploratory study examining how automated writing evaluation (AWE) can inform our understanding of the relationship between postsecondary writing skill and broader indicators of college success. The exploratory study presented in this paper was conducted using test-taker essays from a standardized writing assessment of postsecondary student learning outcomes. Findings showed that for the essays, AWE features were found to be predictors of broader outcomes measures: college success indicators and learning outcomes measures. Study findings expose AWE's potential to support educational analytics --i.e., relationships between writing skill and broader outcomes -moving AWE beyond writing assessment and instructional use cases.

show abstract

Assessing Civic and Intercultural Competency in Higher Education: The ETS HEIghten® Approach

Liu

Roohr

Rios

2018

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

Economic globalization and interdisciplinary advancements have increased the demand for college graduates to possess transferable skills that would allow them to contribute effectively to the modern workforce. In particular, transferable competencies such as civic competency and intercultural competency are critical for colleges to prepare responsible citizens and productive workers. Despite the recognized importance, the choice and quality of assessment for such competencies have been fairly limited due to the challenges in defining such complex, multidimensional constructs and identifying item types that can adequately assess them. In this report, we describe the principles we followed to operationalize definitions for civic competency and intercultural competency and the process we followed to design assessments for these 2 competencies. Findings from a large‐scale pilot test are reported. Results showed that these multidimensional constructs can be adequately assessed and that there is room for students to improve in these areas. Implications for higher education institutions on how to promote these critical competencies are discussed.

show abstract

Assessing critical thinking in higher education: the HEIghten™ approach and preliminary validity evidence

Cited by 76 publications

References 18 publications

Preliminary Evaluation of the Psychometric Quality of HEIghtenTM Quantitative Literacy

Preliminary Evaluation of the Psychometric Quality of HEIghtenTM Quantitative Literacy

Exploring Relationships Between Writing and Broader Outcomes With Automated Writing Evaluation

Assessing Civic and Intercultural Competency in Higher Education: The ETS HEIghten® Approach

Contact Info

Product

Resources

About