Extracting Human Face Similarity Judgments: Pairs or Triplets?

Li, Linjie; Song, Amanda; Malave, Vicente L.; Cottrell, Garrison W.; Yu, Angela J.

doi:10.1167/16.12.719

Cited by 14 publications

(13 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is directly useful in estimating the sample size required for a target level of reliability in studies using these measures. It also gives an indication of how efficiently these measures are able to extract information from responses; this is useful because different methods might take different numbers of trials to produce reliable answers (Li et al 2016;Sprouse & Almeida 2017a).…”

Section: Measure Evaluationmentioning

confidence: 99%

Quantifying sentence acceptability measures: Reliability, bias, and variability

Langsford

Perfors

Hendrickson

et al. 2018

Glossa: A Journal of General Linguistics

View full text Add to dashboard Cite

Understanding and measuring sentence acceptability is of fundamental importance for linguists, but although many measures for doing so have been developed, relatively little is known about some of their psychometric properties. In this paper we evaluate within-and between-participant test-retest reliability on a wide range of measures of sentence acceptability. Doing so allows us to estimate how much of the variability within each measure is due to factors including participantlevel individual differences, sample size, response styles, and item effects. The measures examined include Likert scales, two versions of forced-choice judgments, magnitude estimation, and a novel measure based on Thurstonian approaches in psychophysics. We reproduce previous findings of high between-participant reliability within and across measures, and extend these results to a generally high reliability within individual items and individual people. Our results indicate that Likert scales and the Thurstonian approach produce the most stable and reliable acceptability measures and do so with smaller sample sizes than the other measures. Moreover, their agreement with each other suggests that the limitation of a discrete Likert scale does not impose a significant degree of structure on the resulting acceptability judgments.

show abstract

Section: Measure Evaluationmentioning

confidence: 99%

Quantifying sentence acceptability measures: Reliability, bias, and variability

Langsford

Perfors

Hendrickson

et al. 2018

Glossa: A Journal of General Linguistics

View full text Add to dashboard Cite

show abstract

“…In our first study we compare these different models to human performance in a triadic comparison task, in which people were are asked to pick the most related pair out of three words. Compared to pairwise judgments using rating scales, the triadic task has several advantages: humans find relative judgments easier, it avoids scaling issues, and it leads to more consistent results (Li, Malave, Song, & Yu, 2016). In previous work we have shown that this task can be used for a wide range of semantic comparisons between Dutch words : people make consistent judgments regardless of whether all of the words involved belong to basic categories (lemon -limeapple), share a domain (drill -violinbicycle), or are weakly related (Sundayvitaminidiot).…”

Section: Study 1: Basic Level Triadic Comparisonsmentioning

confidence: 99%

Visual and Affective Grounding in Language and Mind

Deyne¹,

Navarro²,

Collell³

et al. 2018

Preprint

View full text Add to dashboard Cite

One of the main limitations in natural language-based approaches to meaning is that they are not grounded. In this study, we evaluate how well different kinds of models account for people’s representations of both concrete and abstract concepts. The models are both unimodal (language-based only) models and multimodal distributional semantic models (which additionallyincorporate perceptual and/or affective information). The language-based models include both external (based on text corpora) and internal (derived from word associations) language. We present two new studies and a re-analysis of a series of previous studies demonstrating that the unimodal performance is substantially higher for internal models, especially when comparisons at the basiclevel are considered. For multimodal models, our findings suggest that additional visual and affective features lead to only slightly more accurate mental representations of word meaning than what is already encoded in internal language models; however, for abstract concepts, visual andaffective features improve the predictions of external text-based models. Our work presents new evidence that the grounding problem includes abstract words as well and is therefore more widespread than previously suggested. Implications for both embodied and distributional views arediscussed.

show abstract

Section: Measure Evaluationmentioning

confidence: 99%

A systematic comparison and reliability analysis of formal measures of sentence acceptability

Langsford¹,

Hendrickson²,

Perfors³

et al. 2019

Preprint

View full text Add to dashboard Cite

Understanding and measuring sentence acceptability is of fundamental importance for linguists, but although many measures for doing so have been developed, relatively little is known about some of their psychometric properties. In this paper we evaluate within- and between-participant test-retest reliability on a wide range of measures of sentence acceptability. Doing so allows us to estimate how much of the variability within each measure is due to factors including participant-level individual differences, sample size, response styles, and item effects. The measures examined include Likert scales, two versions of forced-choice judgments, magnitude estimation, and a novel measure based on Thurstonian approaches in psychophysics. We reproduce previous findings of high between-participant reliability within and across measures, and extend these results to a generally high reliability within individual items and individual people. Our results indicate that Likert scales and the Thurstonian approach produce the most stable and reliable acceptability measures and do so with smaller sample sizes than the other measures. Moreover, their agreement with each other suggests that the limitation of a discrete Likert scale does not impose a significant degree of structure on the resulting acceptability judgments.

show abstract

Extracting Human Face Similarity Judgments: Pairs or Triplets?

Cited by 14 publications

References 0 publications

Quantifying sentence acceptability measures: Reliability, bias, and variability

Quantifying sentence acceptability measures: Reliability, bias, and variability

Visual and Affective Grounding in Language and Mind

A systematic comparison and reliability analysis of formal measures of sentence acceptability

Contact Info

Product

Resources

About