Drawing inspiration from the lateral lines of fish, the inference of flow characteristics via surface-based data has drawn considerable attention. The current approaches often rely on analytical methods tailored exclusively for potential flows or utilize black-box machine learning algorithms to estimate a specific set of flow parameters. In contrast to a black-box machine learning approach, we demonstrate that it is possible to identify certain modes of fluid flow and then reconstruct the entire flow field from these modes. We use dynamic mode decomposition to parametrize complex, dynamic features across the entire flow field. We then leverage deep neural networks to infer the dynamic modes of the pressure and velocity fields within a large, unsteady flow domain, employing solely a time series of pressure measurements collected on the surface of an immersed obstacle. Our methodology is successfully demonstrated to diverse fluid–structure interaction scenarios, including cases with both free oscillations in the wake of a cylinder and forced oscillations of tandem cylinders, demonstrating its versatility and robustness.