This paper presents the results of a joint effort of a group of multimodality researchers and tool developers to improve the interoperability between several tools used for the annotation and analysis of multimodality. Each of the tools has specific strengths so that a variety of different tools, working on the same data, can be desirable for project work. However this usually requires tedious conversion between formats. We propose a common exchange format for multimodal annotation, based on the annotation graph (AG) formalism, which is supported by import and export routines in the respective tools. In the current version of this format the common denominator information can be reliably exchanged between the tools, and additional information can be stored in a standardized way.

In the introduction to the special issue on ‘Instructing embodied knowledge’, we present a general orientation into this growing field of research, providing the relevant background for the individual contributions. The starting point for the endeavor is the basic observation that practical knowledge or ‘knowing-how’ is typically of a procedural, implicit and embodied nature rather than explicit-conceptual. Given this specific nature, we highlight the fact that for transmitting this type of knowledge, instructors make use of specific multimodal practices that are adjusted to these characteristics. The notion of instructional practice furthermore emphasizes that instructing embodied knowledge is a highly collaborative process between learners and their instructors. In order to provide a broad take on the phenomenon, we review both social-interactional as well as cognitive approaches to embodied knowledge and discuss how the procedural and intercorporeal nature of this knowledge may challenge different views. Independent of the specific approach that is chosen, any account of the construction/instruction of embodied knowledge should emphasize that it is essentially (i) a social activity, (ii) involving the deployment of different semiotic resources, and (iii) using different techniques and devices, such as descriptions, directives and demonstrations. Based on a review of the literature and on the papers in the special issue, we identify a set of key questions that may help to shape the agenda for future studies in the field. The questions relate to the temporal-sequential organization of instructions, the continuum between demonstrations and performances, and the relationship between perceptual access, sensation and the acquisition of embodied knowledge.

No abstract

The present paper analyzes the discourse-pragmatic function of introducing Spanish qué ‘what’- interrogatives with the concessive connective pero ‘but’. In some contexts, a pero-preface contributes to the interpretation of the interrogative as the realization of an interactional challenge rather than a request for information (e.g. an information question). We explore the inferential processes by which the peropreface leads to an interpretation of the interrogative as an interactional challenge and try to demonstrate that this challenge function of pero-prefaced qué-interrogatives may not only achieved ‘ad hoc’ by a local combination of the constitutive elements, but also by conventionalized form-function associations that developed diachronically. In a first step, we analyze pero-prefaced qué-interrogatives in a corpus of spoken Present Day Spanish. There are three main functions of pero-prefaces: to signal that a previous answer to the same interrogative is insufficient, to insist on an answer to a previously unattended request, or to challenge an immediately preceding action by an interlocutor. Using methodology from variationist linguistics, we identify entrenched patterns of pero-prefaced qué-interrogatives that have conventionalized the challenge function. In a second step, we conduct a diachronic variationist analysis of the development of Spanish pero-prefaced qué-interrogatives between 1700 and 1975, testing the hypothesis that the challenge reading developed later than the question reading. Our results show that due to their largely monological nature, the same inferential processes cued by pero lead to different discourse functions in historical texts. Over time, however, the use of pero-prefaced interrogatives started to become more likely in constructed dialogues. We argue that this change reflects an ongoing conventionalization of the challenge function in pero-prefaced interrogatives in spoken language.

Demonstrations are a central resource for instructing body knowledge. They allow instructors to provide learners with a structured perceptual access to the performance of an activity. The present paper considers demonstrations as inherently social activities, in which not only the instructor but also the learners may participate. A particular form of co-participation is that learners synchronize their own bodily actions with the demonstration of the instructor. The paper examines two practices of synchronization in demonstrations. In emergent synchronizations the instructor invites the student(s) to synchronize, rather than request them to do so. In orchestrated synchronizations teachers actively pursue the students’ bodily synchronization. The two practices are typically used for different instructional purposes. While emergent synchronizations are typically used in corrective instructions, orchestrated synchronizations are typically used to instruct new knowledge. Based on a large corpus of instructions in dancing Argentine Tango, the paper uses multimodal interaction analysis to characterize both practices regarding their interactional organization, their functional properties and the resources used by the participants to establish synchronization.