Semantic Image Collection Summarization With Frequent Subgraph Mining

Pasini, Andrea; Giobergia, Flavio; Pastor, Eliana; Baralis, Elena

doi:10.1109/access.2022.3229654

Cited by 6 publications

(9 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As scene graphs are widely used for describing visual objects and their relationships for a single image [31], they are also used for describing multiple images. Pasini et al [2] proposed an image-collection summarization method based on frequent subgraph mining and represents an image collection in a subgraph form on the MS-COCO dataset [24]. Yang et al [32] introduced a challenging task, named Panoptic Video Scene Graph Generation (PVSG), which aims to generate a summarized scene graph of real-world data and contributed a new panoptic video dataset for this task.…”

Section: C: Scene Graph Representationmentioning

confidence: 99%

“…Due to the lack of ground truth for this task, we use common metrics that are used in image collection scene-graph summarization tasks [2], [19]; similarity [16], [49], [50], coverage [28], [51], and diversity [52], [53] of a generated scene graph to the ground-truth scene graph of each image. However, most evaluation techniques focus on estimating the generating precision, in which the evaluation score tends to increase based on the quantity of the generated results.…”

Section: Evaluation Processmentioning

confidence: 99%

“…To evaluate the proposed method in the multiple-images summarization setting, we choose three baseline methods; Semantic Image Summarization (SImS) [2], Image Collection Captioning (ICC) [8], [9], and k-Medoids [17]. SImS is a scene graph summarization method on the MS-COCO dataset by finding frequent sub-graphs.…”

Section: ) Baseline For Multiple-images Scene-graph Summarizationmentioning

confidence: 99%

“…However, most existing research focuses on single-image understanding whereas understanding an image collection is still challenging. In recent years, the task of understanding an image collection is focused in various applications [1], such as semantic image retrieval [2], [3], Web-image concept understanding [4], [5], and multiple-image summarization [6]- [9]. Generating a summarized scene graph also shows an advantage in visual storytelling [10], [11] and video summarization [12] applications.…”

Section: Introductionmentioning

confidence: 99%

“…To generate a summarized scene graph of an image collection, a naïve approach would be to do so in the inference phase [18], [19]. However, this is restricted to understanding Comparison between (A) other summarization methods [2], [8], [9], [17]; Scene-graph generation with summarization process and (B) proposed method; End-to-end scene-graph summarization.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Image-Collection Summarization Using Scene-Graph Generation With External Knowledge

Phueaksri,

Kastner,

Kawanishi

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Summarization tasks aim to summarize multiple pieces of information into a short description or representative information. A text summarization task is a task that summarizes textual information into a short description, whereas in an image collection summarization task, also known as the photo album summarization task, the goal is to find the representative visual information of all images in the collection. In recent years, scene-graph generation has shown the advantage of describing the visual contexts of a singleimage, and incorporating external knowledge into the scene-graph generation model has also given effective directions for unseen single-image scene-graph generation. Following this trend, in this paper, we propose a novel scene-graph-based image-collection summarization model. The key idea of the proposed method is to enhance the relation predictor toward relationships between images in an image collection incorporating knowledge graphs as external knowledge for training a model. To evaluate the proposed method, we build an extended annotated MS-COCO dataset for this task and introduce an evaluation process that focuses on estimating the similarity between a summarized scene graph and ground-truth scene graphs. Traditional evaluation focuses on calculating precision and recall scores, which involve true positive predictions without balancing precision and recall. Meanwhile, the proposed evaluation process focuses on calculating the Fscore of the similarity between a summarized scene graph and ground-truth scene graphs which aims to balance both false positives and false negatives. Experimental results show that the use of external knowledge in enhancing the relation predictor achieves better results compared with existing methods.

show abstract

Section: C: Scene Graph Representationmentioning

confidence: 99%

Section: Evaluation Processmentioning

confidence: 99%

Section: ) Baseline For Multiple-images Scene-graph Summarizationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Image-Collection Summarization Using Scene-Graph Generation With External Knowledge

Phueaksri,

Kastner,

Kawanishi

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers

Perotti,

Bertolotto,

Pastor

et al. 2023

Communications in Computer and Information Science

View full text Add to dashboard Cite

An Approach to Generate a Caption for an Image Collection Using Scene Graph Generation

Phueaksri,

Kastner,

Kawanishi

et al. 2023

IEEE Access

View full text Add to dashboard Cite

Summarization is a challenging task that aims to generate a summary by grasping common information of a given set of information. Text summarization is a popular task of determining the topic or generating a textual summary of documents. In contrast, image summarization aims to find a representative summary of a collection of images. However, current methods are still restricted to generating a visual scene graph, tags, and noun phrases, but cannot generate a fitting textual description of an image collection. Thus, we introduce a novel framework for generating a summarized caption of an image collection. Since scene graph generation shows advancement in describing objects and their relationships on a single image, we use it in the proposed method to generate a scene graph for each image in an image collection. Then, we find common objects and their relationships from all scene graphs and represent them as a summarized scene graph. For this, we merge all scene graphs and select part of it by estimating the most common objects and relationships. Finally, the summarized scene graph is input into a captioning model. In addition, we introduce a technique to generalize specific words in the final caption into common concept words incorporating external knowledge. To evaluate the proposed method, we construct a dataset for this task by extending the annotation of the MS-COCO dataset using an image retrieval method. The evaluation of the proposed method on this dataset showed promising performance compared to text summarization-based methods.

show abstract

Semantic Image Collection Summarization With Frequent Subgraph Mining

Cited by 6 publications

References 48 publications

Image-Collection Summarization Using Scene-Graph Generation With External Knowledge

Image-Collection Summarization Using Scene-Graph Generation With External Knowledge

Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers

An Approach to Generate a Caption for an Image Collection Using Scene Graph Generation

Contact Info

Product

Resources

About