A model for usability testing in challenging environments

A Survey of Field-based Testing Techniques

“…Types of oracles Domain-dependent ( [63]; [53]; [24]; [75]; [69]; [22]; [62]; [27]; [95]; [78]; [37]; [2]; [3]; [71]; [70]; [12]; [46]; [72]; [80]; [6]; [38]; [98]; [31]; [18]; [20]; [10]; [51]; [93]; [32]; [8]; [76]; [86]; [41]; [21]; [48]) Specification based ( [21]; [12]; [46]; [86]; [41]; [6]; [20]; [10]; [76])…”

“…User interactions [89]; [69]; [82]; [73]; [47]; [34]; [32] Interactions with other systems ( [74]; [89]; [83]; [99]; [5]; [102])…”

“…Some approaches access logged information, such as operating systems, network, and sensors logs [3,28]. Other approaches propose ad hoc components for monitoring users' or environment's events that trigger reconfiguration [53], [69]. Yet other approaches rely on periodic inspection of attributes of the system and the environment, such as system resource consumption [54] and network data [51], [100].…”

“…Lee et al [67] and Loustarinen et al [69] rely on manual test case selection based on domain knowledge, which may be accurate, but slow and error-prone. Many approaches propose automatic procedures.…”

“…Ye and Jacobson's approach [101] uses a similar orchestration strategy, assuming that instead of coverage information, the testable services espose events that can be monitored through a dedicated interface by the third-party service. Luostarinen et al [69] use field testing for remote usability testing, by requiring to natively integrate a dedicated API within the user interface. Vain et al [95] propose testing in operation for mission-critical systems, with an orchestration strategy based on a model-based conformance testing approach.…”