Reliability of self-rated experience and confidence as predictors for students’ performance in software engineering

Abstract

Students’ experience is used in empirical software engineering research as well as in software engineering education to group students in either homogeneous or heterogeneous groups. To do so, students are commonly asked to self-rate their experience, as self-rated experience has been shown to be a good predictor for performance in programming tasks. Another experience-related measurement is participants’ confidence (i.e., how confident is the person that their given answer is correct). Hence, self-rated experience and confidence are used as selector or control variables throughout empirical software engineering research and software engineering education. In this paper, we analyze data from several student experiments conducted in the past years to investigate whether self-rated experience and confidence are also good predictors for students’ performance in model comprehension tasks. Our results show that while students can somewhat assess the correctness of a particular answer to one concrete question regarding a conceptual model (i.e., their confidence), their overall self-rated experience does not correlate with their actual performance. Hence, the use of the commonly used measurement of self-rated experience as a selector or control variable must be considered unreliable for model comprehension tasks.

Metrics to Estimate Model Comprehension Quality: Insights from a Systematic Literature Review

“…In a recent and comprehensive study by Daun et al [32], the authors collated the results of a series of experiments investigating the role of experience and confidence on model comprehension. In four experiments involving between 55 and 238 undergraduate and graduate students, participants were asked to self-assess their experience and confidence regarding visual notations, exposed to functional design diagrams and message sequence charts, and asked a series of simple questions pertaining to the content of the diagrams.…”

Model inspections in the engineering of collaborative cyber‐physical systems with instance‐level review diagrams

“…We evaluated the use of instance‐level review diagrams to support the inspection of collaborative CPSs by conducting a controlled experiment to compare the usefulness of the different types of diagrams for model inspections. The experiment setup is based on previously published experiments 65 . In addition, to ensure comparability with other empirical studies, the structure of the section is based on the recommendations given in Jedlitschka et al and Wohlin 14,15 …”

“…The experiment setup is based on previously published experiments. 65 In addition, to ensure comparability with other empirical studies, the structure of the section is based on the recommendations given in Jedlitschka et al and Wohlin. 14,15 F I G U R E 7 Two collaborative cyber-physical system (CPS) network configurations with the same communication patterns…”