Extracting Possessions from Social Media: Images Complement Language

Abstract

This paper describes a new dataset and experiments to determine whether authors of tweets possess the objects they tweet about. We work with 5,000 tweets and show that both humans and neural networks benefit from images in addition to text. We also introduce a simple yet effective strategy to incorporate visual information into any neural network beyond weights from pretrained networks. Specifically, we consider the tags identified in an image as an additional textual input, and leverage pretrained word embeddings as usually done with regular text. Experimental results show this novel strategy is beneficial.

Determining Event Outcomes: The Case of #fail

“…Neural Network Architecture. The neural network is inspired by our previous work (Chinnappa et al, 2019) and Cai et al (2019). It includes two components: one for the text and another one for the image (above and below dotted line in Figure 2).…”

Beyond Possession Existence: Duration and Co-Possession

“…Regarding time, they exclusively target possessions that held true when the weblog was written-not before or after. More recently, we investigate the problem of determining whether authors of tweets possess the objects they tweet about, and use tweets consisting of text and images (Chinnappa et al, 2019). All of these previous efforts target possession existence (i.e., whether a possession relation holds true) and very limited temporal information.…”

“…Since existing corpora already annotate possession existence, however, it would be suboptimal. Thus we work with the corpora by Chinnappa and Blanco (2018), Banea andMihalcea (2018), andChinnappa et al (2019), and enhance their possession existence annotations with possession duration and co-possession annotations. These source corpora contain 2,257 possession relations, a relatively small amount.…”

“…The possessor is always the author of a weblog, and the possessee is mentioned in the weblog and can be: (a) a concrete object, e.g., car, notebook; (b) an implicit concrete object associated with an event, e.g., car for driving, cell phone for texting; or (c) an abstract object, e.g., wifi, idea. The corpus by Chinnappa et al (2019) contains 5,000 tweets (text + image). We select 1,100 tweets in which the author (the possessor) possesses a concrete object mentioned in the tweet (the possessee).…”

“…Table 1 presents the label distribution in our corpus. We distinguish between possessions identified in text (Chinnappa and Blanco, 2018;Banea and Mihalcea, 2018), and those identified in tweets consisting of text and an image (Chinnappa et al, 2019 sesses not present in WordNet (e.g., Garmin, dupioni) and those not subsumed by the top 4 most frequent synsets have roughly the same distribution than all possessees (Table 1). Regarding copossession, devices (e.g., computer, watch) and vehicles (e.g., plane, truck) follow a similar distribution: co-possession is roughly twice as likely.…”