Foundations of multimodal representations: a taxonomy of representational modalities

Bernsen, Niels Ole

doi:10.1016/0953-5438(94)90008-6

Cited by 92 publications

(62 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Precise definitions are presented in the next paragraph. (Coutaz and Caelen, 1991), (Maybury, 1993; and (Bernsen, 1994), among others, define 'media' and 'modalities' contrastingly.…”

Section: Multimodality: State Of the Artmentioning

confidence: 99%

“…Bernsen, 1994), our aim being to investigate the possible influence of the type of visual scenes displayed on target selection times and accuracy.…”

Section: Scene Selection Criteriamentioning

confidence: 99%

“…Bernsen, 1994), it is a thorough inventory of the output modalities available to user interface designers.…”

Section: Definitions: Multimodality Versus Multimediamentioning

confidence: 99%

See 2 more Smart Citations

Do Oral Messages Help Visual Search?

Carbonell

Kieffer

2005

Advances in Natural Multimodal Dialogue Systems

View full text Add to dashboard Cite

“…Precise definitions are presented in the next paragraph. (Coutaz and Caelen, 1991), (Maybury, 1993; and (Bernsen, 1994), among others, define 'media' and 'modalities' contrastingly.…”

Section: Multimodality: State Of the Artmentioning

confidence: 99%

“…Bernsen, 1994), our aim being to investigate the possible influence of the type of visual scenes displayed on target selection times and accuracy.…”

Section: Scene Selection Criteriamentioning

confidence: 99%

See 1 more Smart Citation

Do Oral Messages Help Visual Search?

Carbonell

Kieffer

2005

Advances in Natural Multimodal Dialogue Systems

View full text Add to dashboard Cite

“…For example, Bernsen [3] proposed a taxonomy of generic unimodalities consisting of various features. Other scholars studied the so-called media allocation problem (i.e., how to determine which information to allocate to which medium) and tried to identify which factors play a role in media allocation [1].…”

Section: This Disorder Involves Damage To Muscles Tendons and Nervesmentioning

confidence: 99%

Experiments in Multimodal Information Presentation

Hooijdonk¹,

Bosma²,

Krahmer

et al. 2011

Interactive Multi-Modal Question-Answering

View full text Add to dashboard Cite

“…They essentially support the automatic generation of multimodal presentations. Bernsen [6] proposed a modality taxonomy, based on the observation that different modalities have different representational power. This taxonomy identifies modalities by a set of properties, i.e.…”

Section: Related Workmentioning

confidence: 99%

Modality effects on cognitive load and performance in high-load information presentation

Cao

Theune

Nijholt

2009

Proceedings of the 14th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

In this study, we argue that modality planning in multimodal presentation systems needs to consider the modality characteristics at not only the presentational level but also the cognitive level, especially in a situation where the information load is high and the user task is time-critical. As a first step towards automatic cognitive-aware modality planning, we integrated the effect of different modalities on cognitive load and performance, using a high-load information presentation scenario. Mainly based on modality-related psychology theories, we selected five modality conditions (text, image, text+image, text+speech, and text+sound) and made hypotheses about their effects on cognitive load. Modality effects were evaluated by two cognitive load measurements and two performance measurements. Results confirmed most of the predicted modality effects, and showed that these effects become significant when the information load and the task demand are high. The findings of this study suggest that it is highly necessary to encode modality-related principles of human cognition into the modality planning procedure for systems that support high-load human-computer interaction.

show abstract

Foundations of multimodal representations: a taxonomy of representational modalities

Cited by 92 publications

References 2 publications

Do Oral Messages Help Visual Search?

Do Oral Messages Help Visual Search?

Experiments in Multimodal Information Presentation

Modality effects on cognitive load and performance in high-load information presentation

Contact Info

Product

Resources

About