On the Adoption of Empirical Methods and Systematic Reviews in the Brazilian Symposium on Human Factors in Computing Systems

Abstract

Context: Empirical studies (ES) and systematic reviews (SR) play an essential role in the Human-Computer Interaction (HCI) field as its focus is on evaluating the end-user and usability of software solutions and synthesizing the evidence found by the HCI community. Even though the adoption of empirical evaluation techniques and SR has gained popularity in recent years, the consistent use of a methodology is still maturing. Goal: This study aims to provide a qualitative and quantitative assessment of the current status of ES and SR presented in the research papers published at the proceedings of the Brazilian Symposium on Human Factors in Computing Systems (IHC Symposium). Method: We conduct an empirical study on the papers over the 18 editions in the IHC Symposium to answer four research questions. Our study proposes a protocol to identify and assess ES and SR reported in the papers published at the IHC Symposium. Results: From the sample of 259 studies, we find 131 ES and SR (~51%). We have characterized and categorized the ES into case studies, experiments, and surveys. Further, we found evidence that these studies' quantity and quality have been increased over the IHC Symposium editions, and almost half of these studies give detailed information making possible their replication. Conclusion: We hope that each study's characterization can support the conduction of new ES and SR by the HCI Brazilian community, producing more reliable results and reducing or eliminating biases.