Particle tracking velocimetry (PTV) is widely used to measure time-resolved, three-dimensional velocity and pressure fields in fluid dynamics research. Inaccurate localization and tracking of particles is a key source of error in PTV, especially for single camera defocusing, plenoptic imaging, and digital in-line holography (DIH) measurements. To address this, we developed stochastic particle advection velocimetry (SPAV): a statistical data loss that improves the accuracy of PTV. SPAV is based on an explicit particle advection model that predicts particle positions over time as a function of the estimated velocity field. The model can account for non-ideal effects like drag on inertial particles. A statistical data loss that compares the tracked and advected particle positions, accounting for arbitrary localization and tracking uncertainties, is derived and approximated. We implement our approach using a physics-informed neural network, which simultaneously minimizes the SPAV data loss, a Navier–Stokes physics loss, and a wall boundary loss, where appropriate. Results are reported for simulated and experimental DIH-PTV measurements of laminar and turbulent flows. Our statistical approach significantly improves the accuracy of PTV reconstructions compared to a conventional data loss, resulting in an average reduction of error close to 50%. Furthermore, our framework can be readily adapted to work with other data assimilation techniques like state observer, Kalman filter, and adjoint–variational methods.