Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering

Lipping, Samuel; Sudarsanam, Parthasaarathy; Drossos, Konstantinos; Virtanen, Tuomas

doi:10.23919/eusipco55093.2022.9909680

Cited by 14 publications

(4 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Image QA datasets : recent survey on visual QA and visual reasoning [82] provides a full list of images/visual question-answering (VQA) including reasoning tasks. Audio QA datasets : DAQA [83] on audio temporal reasoning, Clotho-AQA [84] on binary and multichoice audio QA. Video QA datasets : such as VideoQA [85] for multi-domain, MovieQA [86]/MovieFIB [87]/TVQA [88]/KnowIT VQA [89] for movies and shows, MarioQA [90] for games, PororoQA [91] for cartoons, TurorialVQA [92] for tutorials, CLEVRER [93] for physical & causal reasoning.…”

Section: Datasetsmentioning

confidence: 99%

Complex QA and language models hybrid architectures, Survey

Daull¹,

Bellot²,

Bruno³

et al. 2023

Preprint

View full text Add to dashboard Cite

This paper provides a survey of the state of the art of hybrid language models architectures and strategies for "complex" question-answering (QA, CQA, CPS). Very large language models are good at leveraging public data on standard problems but once you want to tackle more specific complex questions or problems you may need specific architecture, knowledge, skills, tasks, methods, sensitive data, performance, human approval and versatile feedback... This survey extends findings from the robust community edited research papers BIG, BLOOM and HELM which open source, benchmark and analyze limits and challenges of large language models in terms of tasks complexity and strict evaluation on accuracy (e.g. fairness, robustness, toxicity, ...). It identifies the key elements used with Large Language Models (LLM) to solve complex questions or problems. Recent projects like ChatGPT and GALACTICA have allowed non-specialists to grasp the great potential as well as the equally strong limitations of language models in complex QA. Hybridizing these models with different components could allow to overcome these different limits and go much further. We discuss some challenges associated with complex QA, including domain adaptation, decomposition and efficient multi-step QA, long form QA, non-factoid QA, safety and multi-sensitivity data protection, multimodal search, hallucinations, QA explainability and truthfulness, time dimension. Therefore we review current solutions and promising strategies, using elements such as hybrid LLM architectures, human-in-the-loop reinforcement learning, prompting adaptation, neuro-symbolic and structured knowledge grounding, program synthesis, and others. We analyze existing solutions and provide an overview of the current research and trends in the area of complex QA.

show abstract

Section: Datasetsmentioning

confidence: 99%

Complex QA and language models hybrid architectures, Survey

Daull¹,

Bellot²,

Bruno³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The second stage of audio-language training aims to enhance the versatility and applicability of our model through multi-task learning, involving a variety of datasets tailored to different audio processing tasks. ClothoAQA (Lipping et al, 2022b), with about 1,500 entries, is utilized for refining our model's capabilities in question answering based on audio cues, with each sample enriched by six associated questions sourced through crowdsourcing. The instruction tuning phase also capitalizes on the continued use of WavCaps (Mei et al, 2023), alongside AudioCaps (Kim et al, 2019) and Clotho (Drossos et al, 2020), which together contribute about 453,000 audio samples for audio-text training.…”

Section: Multi-task Fine-tuningmentioning

confidence: 99%

“…AQA For the AQA task, we select open-ended Clotho-AQA (Lipping et al, 2022a) and multiple-choice TUT2017 (Mesaros et al, 2016) dataset as our benchmark and report accuracy as the evaluation metric.…”

Section: Evaluation Benchmarksmentioning

confidence: 99%

TIGER: A Web Portal of Tumor Immunotherapy Gene Expression Resource

chen

Luo

Zhang

et al. 2022

Preprint

View full text Add to dashboard Cite

Immunotherapy is a promising cancer treatment method, but only a few patients benefit from it. Developing new immunotherapy strategies and effective biomarkers of response and resistance is urgent. Recent high-throughput bulk and single-cell gene expression profiling technologies have generated valuable resources in these regards. However, these resources are not well organized and systematic analysis is difficult. Here, we present TIGER, a tumor immunotherapy gene expression resource, which contains bulk transcriptome data of 1508 tumor samples with immunotherapy clinical outcome and 11,057 tumor/normal samples without immunotherapy clinical outcome, as well as single-cell transcriptome data of 2,116,945 immune cells of 655 samples. TIGER provides many useful modules to analyze the collected data and user-provided data. Using the resource in TIGER, we identified a tumor enriched subset of CD4+ T cells. Melanoma patients with higher signature score of this subset have significantly better response and survival under immunotherapy. We believe that TIGER will be helpful for understanding anti-tumor immunity mechanisms and discovering effective biomarkers. TIGER is freely accessible at http://tiger.canceromics.org/.

show abstract

“…More recently, there has been a surge of interest in establishing a more profound comprehension of audio content by connecting audio and language. A number of audio-language (AL) multimodal learning tasks have been introduced, such as textto-audio retrieval [8], [9], automated audio captioning [10]- [12], audio question answering [13], [14], text-based sound generation [15]- [18], and text-based sound separation [19].…”

Section: Introductionmentioning

confidence: 99%

Automated audio captioning: an overview of recent progress and new challenges

Mei

Liu

Plumbley

et al. 2022

J AUDIO SPEECH MUSIC PROC.

View full text Add to dashboard Cite

Automated audio captioning is a cross-modal translation task that aims to generate natural language descriptions for given audio clips. This task has received increasing attention with the release of freely available datasets in recent years. The problem has been addressed predominantly with deep learning techniques. Numerous approaches have been proposed, such as investigating different neural network architectures, exploiting auxiliary information such as keywords or sentence information to guide caption generation, and employing different training strategies, which have greatly facilitated the development of this field. In this paper, we present a comprehensive review of the published contributions in automated audio captioning, from a variety of existing approaches to evaluation metrics and datasets. We also discuss open challenges and envisage possible future research directions.

show abstract

Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering

Cited by 14 publications

References 17 publications

Complex QA and language models hybrid architectures, Survey

Complex QA and language models hybrid architectures, Survey

TIGER: A Web Portal of Tumor Immunotherapy Gene Expression Resource

Automated audio captioning: an overview of recent progress and new challenges

Contact Info

Product

Resources

About