LEAF-QA: Locate, Encode &amp; Attend for Figure Question Answering

Chaudhry, Ritwick; Shekhar, Sumit; Gupta, Utkarsh; Maneriker, Pranav; Bansal, Prann; Joshi, Ajay

doi:10.1109/wacv45572.2020.9093269

Cited by 57 publications

(89 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The VQA models work poorly on such a dataset. Another dataset made with figures and charts of real world data released in 2020 is LEAFQA [95]. This includes 250,000 annotated images with two million question answer pairs.…”

Section: Datasetsmentioning

confidence: 99%

Towards Assisting the Visually Impaired: A Review on Techniques for Decoding the Visual Data From Chart Images

Shahira

Lijiya

2021

IEEE Access

View full text Add to dashboard Cite

The textual data of a document is supplemented by the graphical information in it. To make communication easier, they contain tables, charts and images. However, it excludes a section of our population -the visually impaired. With technological advancements, the blind can access the documents through text to speech software solutions. In this method, even images can be conveyed by reading out the figure captions. However, charts and other statistical comparisons which involve critical information are difficult to be "read" out this way. Aim of this paper is to analyse various methods available to solve this vexatious issue. We survey the state-of-the-art works that do the exact opposite of graphing tools. In this paper, we explore the existing literature in understanding the graphs and extracting the visual encoding from them. We classify these approaches into modality-based approaches, conventional and deep-learning based methods. The survey also contains comparisons and analyses relevant study datasets. As an outcome of this survey, we observe that: (i) All existing works under each category need decoding in a variety of graphs. (ii) Among the approaches, deep learning performs remarkably well in localisation and classification. However, it needs further improvements in reasoning from chart images. (iii) Research works are still in progress to access data from vector images. Recreating data from the raster images has unresolved issues. Based on this study, the various applications of decoding the graphs, challenges and future possibilities are also discussed. This paper explores current works in the extraction of chart data, which seek to enable researchers in Human Computer Interaction to achieve human-level perception of visual data by machines. In this era of visual summarisation of data, the AI approaches can automate the underlying data extraction and hence provide the natural language descriptions to support visually disabled users.

show abstract

Section: Datasetsmentioning

confidence: 99%

Towards Assisting the Visually Impaired: A Review on Techniques for Decoding the Visual Data From Chart Images

Shahira

Lijiya

2021

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Charts Question Answering (CQA) (Kafle et al, 2018;Kahou et al, 2017;Chaudhry et al, 2020;Methani et al, 2020a) is the task designed on the lines of Visual Question Answering (VQA) (Antol et al, 2015;Malinowski and Fritz, 2014) which requires answering natural language questions about the data visualisations such as bar charts, pie charts, etc. The problem provides us with ability to understand charts using natural language queries, as well as grounding to the natural language statements for the reasoning operations being carried out to retrieve the final answer to the query.…”

Section: Introductionmentioning

confidence: 99%

“…Despite data visualisations being ubiquitous in documents, the problem has received sparse attention in the literature. The earlier datasets like DVQA (Kafle et al, 2018) and FigureQA (Kahou et al, 2017) consist of charts generated from synthetic data, though there has been a push for data charts generated from real sources (Chaudhry et al, 2020;Methani et al, 2020a) as well. Due to the problems discussed above, the prior work noted that VQA algorithms cannot be applied directly to CQA.…”

Section: Introductionmentioning

confidence: 99%

“…While FigureQA (Kahou et al, 2017) uses relational networks for question/answering, DVQA (Kafle et al, 2018) combines text detection and VQAbased attention modules to answer chart questions. LEAF-QA (Chaudhry et al, 2020) encodes question/answers in terms of chart elements, to handle infinite vocabulary problem, while resorting to a VQA-based model as the backbone.…”

Section: Introductionmentioning

confidence: 99%

“…• We extend the recently proposed LEAF-QA dataset (Chaudhry et al, 2020) to generate a harder and more balanced dataset.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering

Singh

Shekhar

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

Self Cite

View full text Add to dashboard Cite

Chart Question Answering (CQA) is the task of answering natural language questions about visualisations in the chart image. Recent solutions, inspired by VQA approaches, rely on image-based attention for question/answering while ignoring the inherent chart structure. We propose STL-CQA which improves the question/answering through sequential elements localization, question encoding and then, a structural transformer-based learning approach. We conduct extensive experiments while proposing pre-training tasks, methodology and also an improved dataset with more complex and balanced questions of different types. The proposed methodology shows a significant accuracy improvement compared to the stateof-the-art approaches on various chart Q/A datasets, while outperforming even human baseline on the DVQA Dataset. We also demonstrate interpretability while examining different components in the inference pipeline.

show abstract