<title>Video and image retrieval beyond the cognitive level: the needs and possibilities</title>

Abstract

The worldwide research efforts in the area of image and video retrieval have concentrated so far on increasing the efficiency and reliability of extracting the elements of image and video semantics and so on improving the search and retrieval performance at the cognitive level of content abstraction. At this abstraction level, the user is searching for "factual" or "objective" content such as image showing a panorama of San Francisco, an outdoor or an indoor image, a broadcast news report on a defined topic, a movie dialog between the actors A and B or the parts of a basketball game showing fast breaks, steals and scores. These efforts, however, do not address the retrieval applications at the so-called affective level of content abstraction where the "ground truth" is not strictly defined. Such applications are, for instance, those where subjectivity of the user plays the major role, e.g. the task of retrieving all images that the user "likes most", and those that are based on "recognizing emotions" in audiovisual data. Typical examples are searching for all images that "radiate happiness", identifying all "sad" movie fragments and looking for the "romantic landscapes", "sentimental" movie segments, "movie highlights" or "most exciting" moments of a sport event. This paper discusses the needs and possibilities for widening the current scope of research in the area of image and video search and retrieval in order to enable applications at the affective level of content abstraction.

Video Affective Content Recognition Based on Film Grammars and Fuzzy Evaluation

“…It is reasonable to build fuzzy membership function on affective response. Let i U denotes the scene feature of order ( 1,2) i i = can be interpreted as a measure of our belief that the basic emotion response ( 1, 2,3) j j = will arise in user while watching the scene which corresponds to i u . In order to determine the fuzzy function, ten testing persons are invited to take part in our experiment.…”

“…In fact, the scene is the minimum semantic unit in the video. If the affective content of video scene is detected, users can search the most exciting video segments, or rank, or retrieval the video database by emotions [1] , or analyze affective-level semantic of a video [2] . In short, the analysis of affective content of video scene is helpful to realize the individual multimedia service.…”

Research on Perception-Oriented Image Scene and Emotion Categorization

“…Images play a very important role in people's lives. There are hundreds of GB or TB of digital image production, publishing and share [1]. Vast amounts of data bring people a variety of convenient, but it also brings a great deal of problems at the same time: it is easy to get lost in the multitude of data and difficult to find the information people really need.…”

Affective content detection using HMMs

“…Much of work, however, has focused on simple low level feature level and semantic analysis level [1]. To deal with a user's preferences effectively in video retrieval and video abstraction, it is desirable to detect affective content from video data.…”

“…To deal with a user's preferences effectively in video retrieval and video abstraction, it is desirable to detect affective content from video data. In other words, if affective content is analyzed, a user can retrieve the most interesting video clips or watch most exciting segments of video [1].…”