A user study for comparing the programming efficiency of modifying executable multimodal interaction descriptions: a domain-specific language versus equivalent event-callback code

Abstract

The present paper describes an empirical user study intended to compare the programming efficiency of our proposed domain-specific language versus a mainstream event language when it comes to modify multimodal interactions. By concerted use of observations, interviews, and standardized questionnaires, we managed to measure the completion rates, completion time, code testing effort, and perceived difficulty of the programming tasks along with the perceived usability and perceived learnability of the tool supporting our proposed language. Based on this experience, we propose some guidelines for designing comparative user studies of programming languages. The paper also discusses the considerations we took into account when designing a multimodal interaction description language that intends to be well regarded by its users.

Usability Evaluation of Domain-Specific Languages: A Systematic Literature Review

“…Cuenca et al [10] described an empirical study to compare efficiency from a DSL to a GPL. Evaluation was performed through observations and interviews conducted with a pre-defined questionnaire.…”

“…As an example, Fig 1 A DSL should be easy to use and its use should be more, or at least as, efficient and efficacious as a GPL. However, experience and literature [10] show that developers consider that the learning curve may be too expensive when comparing the use of known GPLs. Hence more studies are necessary to evaluate the usability of DSLs.…”

Usability evaluation framework for domain-specific language

“…The Usa-DSL framework is composed of eleven (11) steps. The steps of the Usa-DSL framework are described next (see Figure 3).…”

“…Step 1 -Evaluators Profiles: in this step the evaluator profile is defined, instruments to identify the evaluator are applied, the evaluator profile is analyzed and a report on that is written [1,4,11,12,16].…”

Benefits of session types for software development

“…It is expected that various evaluation methodologies would need to be used, such as observations, interviews, or standardised questionnaires. Some relevant studies have been presented in [2], to compare the programming efficiency of a proposed language versus a mainstream one, in [12] to measure the effect of a new tool on programming tasks, or in [10] to gain understanding of the usage and adoption of a tool by deploying it among professional programmers for several weeks.…”