Validity: Meaning and Relevancy in Assessment for Engineering Education Research

A Dialectic Data Integration Approach for Mixed Methods Survey Validation

“…1 These instruments have significant implications for ensuring high quality research, and the validity of conclusions are often derived solely from quantitative data. Typically, these types of instruments are designed for a specific context and purpose which ties directly to where and when their usage may be considered valid.…”

“…Typically, these types of instruments are designed for a specific context and purpose which ties directly to where and when their usage may be considered valid. 1 In order to ensure that students' scores on such instruments are interpreted accurately in the implementation context, and any action taken as a result of those scores are appropriate and meaningful, survey instruments need to go through rigorous validation procedures. These methods have traditionally been quantitative in nature with a heavy focus on psychometrics.…”

“…Therefore, validation is a dynamic, iterative process that must carefully consider context in addition to theory. 1,6,8,12,14 Messick (1995) acknowledged that all available evidence can and should be utilized when considering the validity of an assessment in a particular context, but also that certain evidence may be stronger than other types. 14 In particular evidence must be introduced to account for six particular aspects of the unified validity construct: content validity, substantive validity, structural validity, generalizability, external aspects of validity, and consequential validity.…”

Challenges and Benefits of Applied Experience as an Engineering Returner in a Ph.D. Program

“…Our development of the GSEMS built on findings from an earlier qualitative study 7 that suggested Eccles' expectancy-value theory was an appropriate framework for understanding returners' achievement-based choices to pursue and persist in an engineering PhD. The instrument development process was also guided by literature on returning students in a variety of disciplines, literature on survey development best practices 31,32,33,34 , and the first-hand experiences of our diverse team as advisors to returners, industry professionals, and a former returner herself. We conducted think-aloud interviews to check for the interpretability and promote greater validity of our initial survey draft and revised it to reflect feedback from these sessions 31 .…”

Shared Leadership in Mechanical Engineering-Centric Capstone Design Teams: A Comparison of Military and Civilian Engineering Programs

“…This leadership model has been in existence for over two decades (see 19 ) and is operationalized with the well-established Multifactor Leadership Questionnaire (MLQ). 20 The Full Range of Leadership model was particularly well suited for the study because of its origins in both military and civilian settings 21 which allowed for construct validity 22 within both military and civilian design teams. The leadership factors that comprise the model were originally developed through interviews with a combination of high ranking military officers and corporate leaders.…”