AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning

Abstract

LLM chains enable complex tasks by decomposing work into a sequence of sub-tasks. Crowdsourcing workflows similarly decompose complex tasks into smaller tasks for human crowdworkers. Chains address LLM errors analogously to the way crowdsourcing workflows address human error. To characterize opportunities for LLM chaining, we survey 107 papers across the crowdsourcing and chaining literature to construct a design space for chain development. The design space connects an LLM designer's objectives to strategies they can use to achieve those objectives, and tactics to implement each strategy. To explore how techniques from crowdsourcing may apply to chaining, we adapt crowdsourcing workflows to implement LLM chains across three case studies: creating a taxonomy, shortening text, and writing a short story. From the design space and our case studies, we identify which techniques transfer from crowdsourcing to LLM chaining and raise implications for future research and development.

AGQA 2.0: An Updated Benchmark for Compositional Spatio-Temporal Reasoning

“…We report the experimental results on the new AGQA 2.0 dataset. Please refer to the original AGQA paper for more details on the Human baseline, the Most Likely baseline, the categories on which we evaluate, and the three evaluated models (HCRN, HME, and PSAC) [2]. Table 1.…”

“…As many visual events are a composition of actors interacting with objects over time, computer vision researchers have developed benchmarks to measure models' ability to reason compositionally. Action Genome Question Answering (AGQA) measures compositional reasoning using a Visual Question Answering (VQA) task [2]. AGQA generates questions about videos using natural language templates and ground truth scene graph annotations.…”

“…Each question in AGQA has a reasoning, semantic, and structural categorization. These categories are described in detail in the original paper [2]. The distribution of the structural types differs from the original paper because we use a stricter balancing algorithm for sequencing questions.…”

“…AGQA splits its questions into structural, semantic, and reasoning subgroups for finegrained analysis. Descriptions of these categories can be found in the original paper [2]. The updated sizes of each category can be found in Table 2, and the number of questions associated with each template can be found in Table 3.…”

“…

Prior benchmarks have analyzed models' answers to questions about videos in order to measure visual compositional reasoning. Action Genome Question Answering (AGQA) is one such benchmark [2]. AGQA provides a training/test split with balanced answer distributions to reduce the effect of linguistic biases.

…”