Talk and Gesture as Process Data

Maddox, Bryan

doi:10.1080/15366367.2017.1392821

Cited by 14 publications

(14 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another strategy for supporting the interpretation of a process indicator derived from log data is to triangulate it with other methods and data sources obtained from a particular cognitive assessment. Here, the process indicator is related to other measures of the same attribute of the work process, for instance, measures based on think-aloud protocols, eye-tracking or video recordings of the test-taker and the screen (Maddox, 2017). These additional measures should already be validated to some extent and interpretable in terms of the target attribute.…”

Section: Nomothetic Spanmentioning

confidence: 99%

From byproduct to design factor: on validating the interpretation of process indicators based on log data

Goldhammer

Hahnel

Kroehne

et al. 2021

Large-scale Assess Educ

View full text Add to dashboard Cite

International large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers’ interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker’s work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation.

show abstract

Section: Nomothetic Spanmentioning

confidence: 99%

From byproduct to design factor: on validating the interpretation of process indicators based on log data

Goldhammer

Hahnel

Kroehne

et al. 2021

Large-scale Assess Educ

View full text Add to dashboard Cite

show abstract

“…Empirical research has shown that in combination with response patterns, RTs can lend valuable insight into interesting test-taker, item and test characteristics, such as pre-knowledge of items, motivation, time-pressure or differential speededness (Bridgeman and Cline, 2004; Wise and Kong, 2005; Meijer and Sotaridona, 2006; van der Linden et al, 2007; van der Linden and Guo, 2008; Marianti et al, 2014; Qian et al, 2016). New types of process data have been explored lately that carry the potential to lend additional insight into (latent) response processes and to improve inferences about constructs of interest (e.g., Azevedo, 2015; He et al, 2016; Goldhammer and Zehner, 2017; Maddox, 2017). To make valid inferences from process data, innovative joint models are needed that are capable of utilizing test-taker data beyond RAs and RTs, while accounting for complex relationships in multiple data types.…”

Section: Introductionmentioning

confidence: 99%

Bayesian Covariance Structure Modeling of Responses and Process Data

Klotzke

Fox

2019

Front. Psychol.

View full text Add to dashboard Cite

A novel Bayesian modeling framework for response accuracy (RA), response times (RTs) and other process data is proposed. In a Bayesian covariance structure modeling approach, nested and crossed dependences within test-taker data (e.g., within a testlet, between RAs and RTs for an item) are explicitly modeled. The local dependences are modeled directly through covariance parameters in an additive covariance matrix. The inclusion of random effects (on person or group level) is not necessary, which allows constructing parsimonious models for responses and multiple types of process data. Bayesian Covariance Structure Models (BCSMs) are presented for various well-known dependence structures. Through truncated shifted inverse-gamma priors, closed-form expressions for the conditional posteriors of the covariance parameters are derived. The priors avoid boundary effects at zero, and ensure the positive definiteness of the additive covariance structure at any layer. Dependences of categorical outcome data are modeled through latent continuous variables. In a simulation study, a BCSM for RAs and RTs is compared to van der Linden's hierarchical model (LHM; van der Linden, 2007 ). Under the BCSM, the dependence structure is extended to allow variations in test-takers' working speed and ability and is estimated with a satisfying performance. Under the LHM, the assumption of local independence is violated, which results in a biased estimate of the variance of the ability distribution. Moreover, the BCSM provides insight in changes in the speed-accuracy trade-off. With an empirical example, the flexibility and relevance of the BCSM for complex dependence structures in a real-world setting are discussed.

show abstract

“…The method that informs this paper is microethnographic as its concern is with faceto-face interaction (talk and gesture) in real-life testing situations rather than the wider social and institutional context of the assessment (see Maddox, 2015;Maddox, 2017). The approach can be described as inductive since it seeks to observe social phenomenon, and to identify patterns that can inform explanation and the development of hypotheses.…”

Section: Methodsmentioning

confidence: 99%

“…Observing testing situations produces detailed process data on real-life assessment practice (Maddox and Zumbo, 2017;Maddox, 2017). This includes ethnographic descriptions of talk and gesture, located in the wider context of the testing situation.…”

Section: Methodsmentioning

confidence: 99%

Interviewer-respondent interaction and rapport in PIAAC

Maddox¹

2018

QAE

Self Cite

View full text Add to dashboard Cite

The paper investigates small-scale, qualitative observations of interviewerrespondent interaction in the Organisation for Economic Cooperation and Development's Programme for the International Assessment of Adult Competencies (PIAAC). Design/methodology/approach: The paper employs video-ethnographic methods to document talk and gesture in assessment in Slovenian household settings. It presents an indepth case study of interaction in a single testing situation. Findings: Observing interaction in assessment captures data on assessment performance that is not available in quantitative analysis of assessment response processes. The character of interviewer-respondent interaction and rapport is shaped by the cognitive demands of assessment and the distinctive ecological setting of the household. Research limitations/implications: Observational data on assessment response processes and interaction in real-life assessments can be integrated into and synthesized with other sources of "process data." Practical implications: Assessment programs such as PIAAC should consider the significance of the household setting on assessment quality and observations of interaction in assessment as a valid source of paradata. 2 Social implications: There is a place for small-scale observational studies of assessment to inform public understanding of assessment quality and validity. Originality/value: The paper provides qualitative insights into the significance of interaction and "interviewer effects" in household assessment settings.

show abstract

Talk and Gesture as Process Data

Cited by 14 publications

References 44 publications

From byproduct to design factor: on validating the interpretation of process indicators based on log data

From byproduct to design factor: on validating the interpretation of process indicators based on log data

Bayesian Covariance Structure Modeling of Responses and Process Data

Interviewer-respondent interaction and rapport in PIAAC

Contact Info

Product

Resources

About