A Replication Study on Code Comprehension and Expertise using Lightweight Biometric Sensors

Fucci, Davide; Girardi, Daniela; Novielli, Nicole; Quaranta, Luigi; Lanubile, Filippo

doi:10.1109/icpc.2019.00050

Cited by 27 publications

(23 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The device we use to acquire the biofeedback is the Empatica E4 3 wristband. We selected it as it is used in several studies in affective computing (Greene et al, 2016) as well as in the field of software engineering (Müller and Fritz, 2015;Fucci et al, 2019)). Using the Empatica E4, we collected the following signals:…”

Section: Biofeedback Device and Signalsmentioning

confidence: 99%

Using Voice and Biofeedback to Predict User Engagement during Requirements Interviews

Ferrari¹,

Huichapa²,

Spoletini³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Capturing users’ engagement is crucial for gathering feedback about the features of a software product. In a market-driven context, current approaches to collect and analyze users’ feedback are based on techniques leveraging information extracted from product reviews and social media. These approaches are hardly applicable in bespoke software development, or in contexts in which one needs to gather information from specific users. In such cases, companies need to resort to face-to-face interviews to get feedback on their products. In this paper, we propose to utilize biometric data, in terms of physiological and voice features, to complement interviews with information about the engagement of the user on the discussed product-relevant topics. We evaluate our approach by interviewing users while gathering their physiological data (i.e., biofeedback) using an Empatica E4 wristband, and capturing their voice through the default audio-recorder of a common laptop. Our results show that we can predict users' engagement by training supervised machine learning algorithms on biometric data, and that voice features alone can be sufficiently effective. The performance of the prediction algorithms are maximised when pre-processing the training data with the synthetic minority oversampling technique (SMOTE). The results of our work suggest that biofeedback and voice analysis can be used to facilitate prioritization of requirements oriented to product improvement, and to steer the interview based on users' engagement. Furthermore, the usage of voice features can be particularly helpful for emotion-aware requirements elicitation in remote communication, either performed by human analysts or voice-based chatbots.

show abstract

Section: Biofeedback Device and Signalsmentioning

confidence: 99%

Using Voice and Biofeedback to Predict User Engagement during Requirements Interviews

Ferrari¹,

Huichapa²,

Spoletini³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is not the norm, however, as some studies prescribe the time the subjects have to complete the comprehension task [2,28,40]. A more recent trend in the field of program comprehension research is the usage of physiological measures by employing fMRI scanners [14,27], biometrics sensors [15,16,44] or eye-tracking devices [15,36]. Lastly, some studies measure perceived understandability [30] which we explicitly distinguish from the other measures.…”

Section: Source Code Understandabilitymentioning

confidence: 99%

An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability

Barón

Wyrich

Wagner

2020

Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

View full text Add to dashboard Cite

Background: Developers spend a lot of their time on understanding source code. Static code analysis tools can draw attention to code that is difficult for developers to understand. However, most of the findings are based on non-validated metrics, which can lead to confusion and code that is hard to understand not being identified. Aims: In this work, we validate a metric called Cognitive Complexity which was explicitly designed to measure code understandability and which is already widely used due to its integration in wellknown static code analysis tools. Method: We conducted a systematic literature search to obtain data sets from studies which measured code understandability. This way we obtained about 24,000 understandability evaluations of 427 code snippets. We calculated the correlations of these measurements with the corresponding metric values and statistically summarized the correlation coefficients through a meta-analysis. Results: Cognitive Complexity positively correlates with comprehension time and subjective ratings of understandability. The metric showed mixed results for the correlation with the correctness of comprehension tasks and with physiological measures. Conclusions: It is the first validated and solely code-based metric which is able to reflect at least some aspects of code understandability. Moreover, due to its methodology, this work shows that code understanding is currently measured in many different ways, which we also do not know how they are related. This makes it difficult to compare the results of individual studies as well as to develop a metric that measures code understanding in all its facets. CCS CONCEPTS • Software and its engineering → Empirical software validation.

show abstract

“…The adoption of EEG signals has been relevant to analyze aspects of human factors in software engineering (Crk & Kluthe, 2016;Fucci et al, 2019;Lee et al, 2018). The electroencephalogram (EEG) is responsible for capturing the electrical signals generated through the interaction of neurons (Cohen, 2017).…”

Section: Introductionmentioning

confidence: 99%

“…The electroencephalogram (EEG) is responsible for capturing the electrical signals generated through the interaction of neurons (Cohen, 2017). Software engineering researchers have been using this indicator along with machine learning techniques to classify task difficulty (Lee et al, 2018), level of experience (Crk & Kluthe, 2016), and categories of programming tasks artifacts (Fucci et al, 2019), i.e., between textual prose and programming language. The EEG signals contain noises such as abnormal readings, electrical interference, and eye movements, which are not of interest to the analysis and therefore contribute to the classifiers' lack of precision as they are not related to a specific cognitive process required in such experiments.…”

Section: Introductionmentioning

confidence: 99%

“…However, despite the evident benefit of filtering EEG signals in studies focused on research areas such as signal processing and clinical image, it remains unknown if filtering an EEG signal can affect the applications focused on software engineering (Crk & Kluthe, 2016;Fucci et al, 2019;Lee et al, 2016;Lee et al, 2018;Radevski et al, 2015). Such research generally claims that filters in EEG signals improve the analysis and helps to obtain more effective results.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The effects of applying filters on EEG signals for classifying developers’ code comprehension

Gonçales

Farias

Kupssinskü

et al. 2021

JART

View full text Add to dashboard Cite

EEG signals are a relevant indicator for measuring aspects related to human factors in Software Engineering. EEG is used in software engineering to train machine learning techniques for a wide range of applications, including classifying task difficulty, and developers’ level of experience. The EEG signal contains noise such as abnormal readings, electrical interference, and eye movements, which are usually not of interest to the analysis, and therefore contribute to the lack of precision of the machine learning techniques. However, research in software engineering has not evidenced the effectiveness when applying these filters on EEG signals. The objective of this work is to analyze the effectiveness of filters on EEG signals in the software engineering context. As literature did not focus on the classification of developers’ code comprehension, this study focuses on the analysis of the effectiveness of applying EEG filters for training a machine learning technique to classify developers' code comprehension. A Random Forest (RF) machine learning technique was trained with filtered EEG signals to classify the developers' code comprehension. This study also trained another random forest classifier with unfiltered EEG data. Both models were trained using 10-fold cross-validation. This work measures the classifiers' effectiveness using the f-measure metric. This work used the t-test, Wilcoxon, and U Mann Whitney to analyze the difference in the effectiveness measures (f-measure) between the classifier trained with filtered EEG and the classifier trained with unfiltered EEG. The tests pointed out that there is a significant difference after applying EEG filters to classify developers' code comprehension with the random forest classifier. The conclusion is that the use of EEG filters significantly improves the effectivity to classify code comprehension using the random forest technique.

show abstract

A Replication Study on Code Comprehension and Expertise using Lightweight Biometric Sensors

Cited by 27 publications

References 45 publications

Using Voice and Biofeedback to Predict User Engagement during Requirements Interviews

Using Voice and Biofeedback to Predict User Engagement during Requirements Interviews

An Empirical Validation of Cognitive Complexity as a Measure of Source Code Understandability

The effects of applying filters on EEG signals for classifying developers’ code comprehension

Contact Info

Product

Resources

About