We marry two powerful ideas: deep representation learning for visual recognition and language understanding, and symbolic program execution for reasoning. Our neural-symbolic visual question answering (NS-VQA) system first recovers a structural scene representation from the image and a program trace from the question. It then executes the program on the scene representation to obtain an answer. Incorporating symbolic structure as prior knowledge offers three unique advantages. First, executing programs on a symbolic space is more robust to long program traces; our model can solve complex reasoning tasks better, achieving an accuracy of 99.8% on the CLEVR dataset. Second, the model is more data-and memory-efficient: it performs well after learning on a small number of training data; it can also encode an image into a compact representation, requiring less storage than existing methods for offline question answering. Third, symbolic program execution offers full transparency to the reasoning process; we are thus able to interpret and diagnose each execution step.

We introduce an alternative scheme for creating topological polaritons (topolaritons) by exploiting the presence of photonic Dirac cones in photonic crystals with triangular lattice symmetry. As recently proposed, topolariton states can emerge from a coupling between photons and excitons combined with a periodic exciton potential and a magnetic field to open up a topological gap. We show that in photonic crystals the opening of the gap can be substantially simplified close to photonic Dirac points. Coupling to Zeeman-split excitons breaks time reversal symmetry and allows to gap out the Dirac cones in a nontrival way, leading to a topological gap similar to the strength of the periodic exciton potential. Compared to the original topolariton proposal [T. Karzig et al., Phys. Rev. X 5, 031001 (2015)], this scheme significantly increases the size of the topological gap over a wide range of parameters. Moreover, the gap opening mechanism highlights an interesting connection between topolaritons and the scheme of [F. D. M. Haldane and S. Raghu, Phys. Rev. Lett. 100, 013904 (2008)] to create topological photons in magneto-optically active materials.

Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not directly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset 1 . For a given set of objects, ComPhy includes few videos of them moving and interacting under different initial conditions. The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions posted on one of the videos. Evaluation results of several state-of-the-art video reasoning models on ComPhy show unsatisfactory performance as they fail to capture these hidden properties. We further propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual perception, physical property learning, dynamic prediction, and symbolic execution into a unified framework. CPL can effectively identify objects' physical properties from their interactions and predict their dynamics to answer questions.Recent studies have established a series of benchmarks to evaluate and diagnose machine learning systems in various physics-related environments (

No abstract