Content-oriented multimedia document understanding through cross-media correlation

Video Text Detection

“…A more systematical and detailed introduction to the discussed techniques may be found in the references for image processing and machine vision in [48][49][50][51]. …”

“…Essentially, multimodal video data originated from the same source tend to be correlated [1,2,48]. It means that different modalities can take a complementary role on solving video content analysis tasks, and the presence of one modality can help understand certain semantics of others.…”

“…To evaluate scene digit recognition algorithms, Netzer et al [46] build the Street View House Numbers (SVHN) dataset to detect and read house-number signs in street view scenes. The SVHN dataset is obtained from a large number of street view scene images [47] using a combination of automated algorithms and the Amazon Mechanical Turk framework [48], which is used to localize and transcribe the single digits. A very large set of images from urban areas in various countries are downloaded.…”

Context-based environmental audio event recognition for scene understanding

“…For example, the latent semantic analysis derived from language processing is proposed as an interesting solution to learn overlapped audio events [47]. Lu et al [22] propose a multimodal correlation network, in which audio-to-audio retrievals can be improved by incorporating visual image information. However, this area requires more studying to apply the techniques efficiently into auditory scene understanding.…”

“…In this paper, we propose a novel audio event recognition framework for acoustic scene understanding based on our previous work on sound classification [3,19], audio summarization [20,21] and audio-visual correlation [4,22]. The term auditory scene here refers to the acoustic modeling of a specific location or site such as home, bus station, restaurant and shopping mall, which is similar to what an image of the same location provides visually.…”