Research experiences instrument: Validation evidence for an instrument to assess the research experiences of engineering PhD students' professional practice opportunities

Abstract

Background There are long‐held concerns about how graduate research programs prepare engineering PhD students for professional practice. Suitable instruments are lacking to effectively assess how research experiences contribute to the success of graduate students becoming professionals. Purpose The purpose of this work is to examine evidence of internal reliability and validity of using the research experiences instrument (REI) scores as a measure of engineering PhD students' professional practice opportunities in their research experiences. Method REI was constructed using an ontological framework. REI was administered twice to engineering PhD students, once to a single university (n = 236) and once to multiuniversities (n = 215). Psychometric analyses were conducted related to validity and reliability evidence, including exploratory and confirmatory factor analyses, and group score comparisons between genders, race/ethnicity, and engineering disciplines. Results Results of both factor analyses aligned with the theoretical five‐factor structure, with a second‐order Opportunity factor. Mean scores were the same between women and men, and for three engineering disciplines, but significantly lower for racially/ethnically minoritized groups. Factor scores indicate that students often lack opportunities to engage with professionals, a likely cause of students' professional practice struggles. Conclusions Evidence of validity has been provided to justify the use of REI to assess the unique research experiences of engineering PhD students as these relate to preparedness for professional practice. REI scores can be used for gender and race/ethnicity comparisons and are generalizable across engineering disciplines. Where students' REI scores show a lack of opportunities, remedial strategies can be implemented.

Measuring the Authenticity of Engineering Learning in Community of Practice: An Instrument Development and Validation

“…Briefly, the purposes of this paper are twofold: Firstly, this paper will make clear the operational definitions for authentic engineering and put forward some appropriate items to form an instrument; Secondly, this research will study the validity and internal consistency of the instrument we have developed. The development of the instrument followed recommendations for instrument development by Netemeyer et al [40], focused on construct design and initial evidence of validity [41].…”

“…Secondly, we referred to Holloway et al work [6], [41], [54], [55] in terms of the process and method of developing a valid and credible scale. But we didn't perform group analysis among different groups (such as different engineering disciplines, different degrees, etc.)…”

“…Engineers are entrusted by the public to apply their professional knowledge and competencies to innovate, design, and implement solutions for societal needs [1]. But in recent years, many engineering students have reported they have difficulties in applying their professional engineering knowledge and skills in real-world contexts [2]- [4], and lacking preparation for solving engineering workplace problems [5], [6], which has become global issues for engineering education. Some higher engineering education institutions have made steps to overcome these challenges.…”

Changing the Conversation Surrounding Students’ Professional Skills: Making the Case for the Importance of Professional Skills, and More Inclusive Language

“…However, graduate students often find it difficult to be a professional through their development as students [28]. The difficulty generally occurs in transferring specific knowledge and skills obtained as students to real-world application and with being sufficiently prepared to enter into professional practice [29,30].…”

“…Importantly, three different theoretical frameworks have considered this challenge to student development [30,31] and were applied to the study's review of student development. First, Säljö's [32] sociocultural framework proposed that students have limited social or cultural understanding of the meaning of being a professional.…”