Web news categorization using a cross-media document graph

Abstract

In this paper we propose a multimedia categorization framework that is able to exploit information across different parts of a multimedia document (e.g., a Web page, a PDF, a Microsoft Office document). For example, a Web news page is composed by text describing some event (e.g., a car accident) and a picture containing additional information regarding the real extent of the event (e.g., how damaged the car is) or providing evidence corroborating the text part. The framework handles multimedia information by considering not only the document's text and images data but also the layout structure which determines how a given text block is related to a particular image. The novelties and contributions of the proposed framework are: (1) support of heterogeneous types of multimedia documents; (2) a documentgraph representation method; and (3) the computation of crossmedia correlations. Moreover, we applied the framework to the tasks of categorising Web news feed data, and our results show a significant improvement over a single-medium based framework.

On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval

“…In parallel with these developments, advances have been reported in multimodal retrieval systems [8], [9], [10], [11], [12], [13], [27]. These are extensions of the classic unimodal systems, where a common retrieval system integrates information from various modalities.…”

“…Kðx; yÞ ¼ expð (7) and (11). Using this architecture, a final round of experiments was used to determine the best combination of 1) BOW codebook size for image representation, 2) number of LDA topics for text representation, and 3) number of KCCA components, for each of the CM, SM, and SCM retrieval regimes and data set.…”

“…This paradigm is of limited use in the modern information landscape, where multimedia content is ubiquitous. Due to this, multimodal modeling, representation, and retrieval have been extensively studied in the multimedia literature [4], [5], [6], [7], [8], [9], [10], [11]. In multimodal retrieval systems, queries combining multiple content modalities (e.g., images and sound of a music video clip) are used to retrieve database entries with the same combination of modalities (e.g., other music video clips).…”