EventMask: A Game-Based Framework for Event-Saliency Identification in Images

Abstract

The concept of "event" emerged in the last years as a key feature to efficiently index and retrieve media. Several approaches have been proposed to analyze the relationship between events and related media, enable event discovery, perform event-based media tagging, indexing, and retrieval. Despite the outstanding work done by several researchers in this area, a major problem that still remains open is how to infer the inherent link between visual concepts and events. In particular, the possibility of understanding which perceptual elements allow a human recognizing the event depicted by an image would for sure open new directions in event media discovery. In this paper we introduce the concept of Event Saliency to define the above event-revealing perceptual elements, and we propose an original method to detect it by exploiting crowd knowledge through gamification. We propose an adversarial game with a hidden purpose, where users are engaged in two competitive roles: masking photos to prevent competitors recognizing the related event, and discovering events in photos masked by other players. Rules and incentives are defined to minimize cheating and force players to focus on details that really matter. Finally, a suitable algorithm composes the masks created by different players on the same media, thus producing a saliency map that, differently from the traditional concept of saliency, does not focus on perceptual prominence but rather on event-related semantics of media. A thorough validation on two public dataset is presented, and initial experiments to apply event saliency in detection tasks are proposed. Furthermore, an event-saliency dataset is disclosed to allow further research in this domain.

Medial Orbitofrontal Cortex, Dorsolateral Prefrontal Cortex, and Hippocampus Differentially Represent the Event Saliency

“…Previously, event saliency has been used to determine the probability of a deviant event in a linguistic experiment (Coulson, King, & Kutas, 1998), but recently, with the availability of videos and advances in computer vision, this term is used for quantifying the magnitude of deviation in a flow of a movie. Event saliency is not determined by changes in a visual scene but relies on following the movies and noting significant changes in the flow of events (Zhang, Han, Jiang, Ye, & Chang, 2017; Rosani, Boato, & De Natale, 2015). Accordingly, event saliency is measured by either asking an independent group to identify the boundaries or, in linguistics, studying the transition from one word to another, which is extracted from a large body of literature.…”

Gamified crowdsourcing: Conceptualization, literature review, and future agenda

“…Altmeyer et al, 2016;Brenner et al, 2014;Carlier et al, 2016;Deng et al, 2016;Dergousoff and Mandryk, 2015;Feyisetan et al, 2015;Itoko et al, 2014;Kobayashi et al, 2015;Kurita et al, 2016;Lessel et al, 2015;Mason et al, 2012;Riegler et al, 2015;Roengsamut et al, 2015;Rosani et al, 2015; Runge et al, 2015 15 Reporting location-based informationBowser et al, 2013; Brito et al, 2015; De Franga et al, 2015; Goncalves et al, 2014; Kawajiri et al, 2014; Y. Liu, Alexandrova, Nakajima et al, 2011*; Martella et al, 2015; Massung et al, 2013; Prandi et al, location tagging, reporting of location-based information, on-location experience, taking location-based photos 2016; Preist et al, 2014; Sheng, 2013; Simões and De Amicis, 2016; Talasila et al, 2016; Uzun et al, 2013 Answering questions/sharing knowledge answering user-generated questions, providing feedback, knowledge-sharing in communities Ipeirotis and Gabrilovich, 2014; Inaba et al, 2015; Y. Liu, Alexandrova, Nakajima et al, 2011*; Machnik et al, 2015; Pothineni et al, 2014; Vasilescu et al, 2014 6 Creative creation work idea creation, algorithm development, requirements elicitation Bentzien et al, 2013; Choi et al, 2014; Dos Santos et al, 2015; Lauto and Valentin, 2016; Snijders et al, 2015; Yakushin and Lee, 2014 6 Text annotation work text annotation, medical text annotation, biological data annotation Cao et al, 2015; Chamberlain, 2014; Dumitrache et al, 2013; Nose and Hishiyama, 2013; Ustalov, 2015 5 Assessment work relationship building, relevance assessment, classification work, decision-making Eickhoff et al, 2012; Harris, 2014; Melenhorst et al, 2015; Prestopnik and Tang, 2015; Yu et al, 2015 5 Searching for and/or optimization of tasks document searching, searching for digital profiles, finding optimal solutionsHe et al, 2014; T. Y. Lee et al, 2013;Nunzio et al, 2016;Sørensen et al, 2016;Tinati et alno clear work description provided, usergenerated tasks, social activitiesCucari et al, 2016;J. J. Lee et al, 2013;Nagai et al, 2014; Sakamoto and Nakajima, 2014 4 References in bold refer to studies in which empirical results about gamification have been reported.…”

“…). Unfortunately, the small amount of studies investigating gamification in the crowdcreating approaches limits the identification of a clear pattern in their gamification implementations Riegler et al, 2015;Roengsamut et al, 2015;Rosani et al 2015;.…”

Event-based media processing and analysis: A survey of the literature

“…There are many studies dealing with video event detection that use intermediate-or high-level video representations, frequently along with a set of low-level features, e.g. [78,40,217,224,191,117,129,109,108,83,82,179,220,118,162].…”