Phy-Q as a measure for physical reasoning intelligence

Abstract

Humans are well versed in reasoning about the behaviours of physical objects and choosing actions accordingly to accomplish tasks, while this remains a major challenge for artificial intelligence. To facilitate research addressing this problem, we propose a new testbed that requires an agent to reason about physical scenarios and take an action appropriately. Inspired by the physical knowledge acquired in infancy and the capabilities required for robots to operate in real-world environments, we identify 15 essential physical scenarios. We create a wide variety of distinct task templates, and we ensure that all the task templates within the same scenario can be solved by using one specific strategic physical rule. By having such a design, we evaluate two distinct levels of generalization, namely local generalization and broad generalization. We conduct an extensive evaluation with human players, learning agents with various input types and architectures, and heuristic agents with different strategies. Inspired by how the human intelligence quotient is calculated, we define the physical reasoning quotient (Phy-Q score) that reflects the physical reasoning intelligence of an agent using the physical scenarios we considered. Our evaluation shows that (1) all the agents are far below human performance, and (2) learning agents, even with good local generalization ability, struggle to learn the underlying physical reasoning rules and fail to generalize broadly. We encourage the development of intelligent agents that can reach the human-level Phy-Q score.

Physics-Based Task Generation through Causal Sequence of Physical Interactions

“…In recent years, researchers have developed various environments as benchmarks and testbeds to evaluate the physical reasoning capabilities of AI agents. These environments are mainly based on tasks that involve taking actions in a physical environment (Xue et al 2023;Bakhtin et al 2019), reasoning about images of physical scenarios (Hong et al 2021;Wolf 2020), and reasoning about videos of physical scenarios (Riochet et al 2020;Yi et al 2020). Among them, task generation methods used in action-based environments are related to this work as we also focus on generating tasks that an agent can interact with and take actions to solve them.…”

“…Among them, task generation methods used in action-based environments are related to this work as we also focus on generating tasks that an agent can interact with and take actions to solve them. Examples of recent action-based environments include Phy-Q (Xue et al 2023), PHYRE (Bakhtin et al 2019), Virtual Tools , OGRE , CausalWorld (Ahmed et al 2021), and RL-Bench (James et al 2020).…”

“…Specifically, only certain points of interest on the target object (such as the ll, ul, and ur) are selected, and it is assumed that the bird is always shot with the full stretch of the slingshot, resulting in only two possible trajectories to reach a target point. Shooting with the full stretch reflects the current capability of Angry Birds playing agents (Xue et al 2023).…”

“…Physics-based puzzles are often utilized to assess the physical reasoning abilities of humans (Diezmann and Watters 2000;Cheke, Loissel, and Clayton 2012), animals (Emery and Clayton 2009), and AI systems (Xue et al 2023;Bakhtin et al 2019). While humans develop the capacity to perform physical reasoning tasks from their infancy (Valenza et al 2006;Baillargeon and DeVos 1991), this has proven to be a challenge for AI systems (Xue et al 2023;Bakhtin et al 2019). With the expanding use of AI systems in physical environments, there is a need for suitable testbeds to enable the advancement of these systems.…”