Towards large-scale multimedia retrieval enriched by knowledge about human interpretation

Shirahama, Kimiaki; Grzegorzek, Marcin

doi:10.1007/s11042-014-2292-8

Cited by 23 publications

(17 citation statements)

References 127 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, this paper complements another survey paper that we have recently published, taking the above-mentioned retrospective approach in the field of multimedia processing (Shirahama & Grzegorzek, 2014). In that survey, we realized that the development of human-machine cooperation requires interdisciplinary expertise such as cognitive science, neuroscience, and ontology engineering.…”

Section: Introductionmentioning

confidence: 67%

“…Consequently, this paper aims to disseminate the problem of human-machine cooperation in LSMR to many researchers in different fields to stimulate interdisciplinary collaborations. To this end, rather than covering various existing methods like Shirahama and Grzegorzek (2014), this paper concentrates on providing intuitive explanations of representative methods. Specifically, the next section focuses on three types of popular machine-based LSMR methods: classical methods using heuristically defined templates, methods that build classifiers using user-provided examples, and their extension in terms of features and classifiers.…”

Section: Introductionmentioning

confidence: 99%

“…In contrast, this paper graphically elaborates core ideas of all these methods (except intuitive manual annotation methods). In the LSMR based on Human-Machine Cooperation section, we also graphically illustrate methods that include not only notable utilization of knowledge about human interpretation, but also fundamental elements to understand the other methods described in Shirahama and Grzegorzek (2014). In addition, considering the research in cognitive science, this section provides several new ideas to achieve novel humanmachine cooperation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Human-Machine Cooperation in Large-Scale Multimedia Retrieval: A Survey

Shirahama¹,

Grzegorzek²,

Indurkhya³

2015

The Journal of Problem Solving

Self Cite

View full text Add to dashboard Cite

Large-Scale Multimedia Retrieval (LSMR) is the task to fast analyze a large amount of multimedia data like images or videos and accurately find the ones relevant to a certain semantic meaning. Although LSMR has been investigated for more than two decades in the fields of multimedia processing and computer vision, a more interdisciplinary approach is necessary to develop an LSMR system that is really meaningful for humans. To this end, this paper aims to stimulate attention to the LSMR problem from diverse research fields. By explaining basic terminologies in LSMR, we first survey several representative methods in chronological order. This reveals that due to prioritizing the generality and scalability for large-scale data, recent methods interpret semantic meanings with a completely different mechanism from humans, though such humanlike mechanisms were used in classical heuristic-based methods. Based on this, we discuss human-machine cooperation, which incorporates knowledge about human interpretation into LSMR without sacrificing the generality and scalability. In particular, we present three approaches to human-machine cooperation (cognitive, ontological, and adaptive), which are attributed to cognitive science, ontology engineering, and metacognition, respectively. We hope that this paper will create a bridge to enable researchers in different fields to communicate about the LSMR problem and lead to a ground-breaking next generation of LSMR systems.

show abstract

Section: Introductionmentioning

confidence: 67%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Human-Machine Cooperation in Large-Scale Multimedia Retrieval: A Survey

Shirahama¹,

Grzegorzek²,

Indurkhya³

2015

The Journal of Problem Solving

Self Cite

View full text Add to dashboard Cite

show abstract

“…Thus, we will adopt features based on more sophisticated distribution models like Gaussian Mixture Model (GMM) [45] and Fisher encoding [40,49]. Especially, these features are represented as very high-dimensional vectors with more than 10,000 dimensions, so that accurate activity recognition can be achieved using a simple and fast classifier like linear SVM.…”

Section: Conclusion and Discussionmentioning

confidence: 99%

On the Generality of Codebook Approach for Sensor-Based Human Activity Recognition

Shirahama

Grzegorzek

2017

Electronics

Self Cite

View full text Add to dashboard Cite

Abstract:With the recent spread of mobile devices equipped with different sensors, it is possible to continuously recognise and monitor activities in daily life. This sensor-based human activity recognition is formulated as sequence classification to categorise sequences of sensor values into appropriate activity classes. One crucial problem is how to model features that can precisely represent characteristics of each sequence and lead to accurate recognition. It is laborious and/or difficult to hand-craft such features based on prior knowledge and manual investigation about sensor data. To overcome this, we focus on a feature learning approach that extracts useful features from a large amount of data. In particular, we adopt a simple but effective one, called codebook approach, which groups numerous subsequences collected from sequences into clusters. Each cluster centre is called a codeword and represents a statistically distinctive subsequence. Then, a sequence is encoded as a feature expressing the distribution of codewords. The extensive experiments on different recognition tasks for physical, mental and eye-based activities validate the effectiveness, generality and usability of the codebook approach.

show abstract

“…As a consequence, the current research focus has moved to larger and more challenging datasets [3,34], such as ImageNet [6]. Such datasets often organize the large number of categories in a hierarchy, according to their semantic belongings.…”

Section: Introductionmentioning

confidence: 99%

CNN-RNN: a large-scale hierarchical image classification framework

Guo¹,

Liu²,

Bakker³

et al. 2017

Multimed Tools Appl

123

View full text Add to dashboard Cite

Objects are often organized in a semantic hierarchy of categories, where finelevel categories are grouped into coarse-level categories according to their semantic relations. While previous works usually only classify objects into the leaf categories, we argue that generating hierarchical labels can actually describe how the leaf categories evolved from higher level coarse-grained categories, thus can provide a better understanding of the objects. In this paper, we propose to utilize the CNN-RNN framework to address the hierarchical image classification task. CNN allows us to obtain discriminative features for the input images, and RNN enables us to jointly optimize the classification of coarse and fine labels. This framework can not only generate hierarchical labels for images, but also improve the traditional leaf-level classification performance due to incorporating the hierarchical information. Moreover, this framework can be built on top of any CNN architecture which is primarily designed for leaf-level classification. Accordingly, we build a high performance network based on the CNN-RNN paradigm which outperforms the original CNN (wider-ResNet) and also the current state-of-the-art. In addition, we investigate how to utilize the CNN-RNN framework to improve the fine category classification when a fraction of the training data is only annotated with coarse labels. Experimental results demonstrate that CNN-RNN can use the coarse-labeled training data to improve the classification of fine categories, and in some cases it even surpasses the performance achieved by fully annotated training data. This reveals that, CNN-RNN can alleviate the challenge of specialized and expensive annotation of fine labels.

show abstract

Towards large-scale multimedia retrieval enriched by knowledge about human interpretation

Cited by 23 publications

References 127 publications

Human-Machine Cooperation in Large-Scale Multimedia Retrieval: A Survey

Human-Machine Cooperation in Large-Scale Multimedia Retrieval: A Survey

On the Generality of Codebook Approach for Sensor-Based Human Activity Recognition

CNN-RNN: a large-scale hierarchical image classification framework

Contact Info

Product

Resources

About