Thematic fit evaluation: an aspect of selectional preferences

Abstract

In this paper, we discuss the human thematic fit judgement correlation task in the context of real-valued vector space word representations. Thematic fit is the extent to which an argument fulfils the selectional preference of a verb given a role: for example, how well "cake" fulfils the patient role of "cut". In recent work, systems have been evaluated on this task by finding the correlations of their output judgements with human-collected judgement data. This task is a representationindependent way of evaluating models that can be applied whenever a system score can be generated, and it is applicable wherever predicate-argument relations are significant to performance in end-user tasks. Significant progress has been made on this cognitive modeling task, leaving considerable space for future, more comprehensive types of evaluation.

A structured distributional model of sentence meaning and processing

“…Differently from other distributional semantic models tested on the thematic fit task, ‘structure’ is now externally encoded in a graph, whose nodes are embeddings, and not directly in the dimension of the embeddings themselves. The fact that the best performing word embeddings in our framework are the Skip-Gram ones is somewhat surprising, and against the finding of previous literature in which bag-of-words models were always described as struggling on this task (Baroni et al 2014; Sayeed et al 2016). Given our results, we also suggested that the dimensionality of the embeddings could be an important factor, much more than the choice of training them on syntactic contexts.…”

“…Another constant finding of previous studies on thematic fit modelling was that high-dimensional, count-based vector representations perform generally better than dense word embeddings, to the point that Sayeed et al (2016) stressed the sensitivity of this task to linguistic detail and to the interpretability of the vector space. Therefore, we tested whether vector dimensionality had an impact on task performance (Table 4).…”

Distributional Models of Word Meaning

“…For example, given the agent butcher, the expected patient of the verb cut is likely to be meat, whereas given the agent coiffeur, the expected patient is likely to be hair. For other research on this topic, see Sayeed et al (2016) and Tilk et al (2016).…”

Modeling Semantic Expectation: Using Script Knowledge for Referent Prediction

“…The similarities are calculated using a Distributional Memory approach similar to that of Baroni and Lenci (2010). Their structured vector space representation has been shown to work well on tasks that evaluate correlation with human thematic fit estimates (Baroni and Lenci, 2010;Baroni et al, 2014;Sayeed et al, 2016) and is thus suited to our task.…”