Context. Understanding how star formation begins and propagates through molecular clouds is a fundamental but still open question. One major difficulty in addressing this question is the lack of precise 3D kinematics and age information of young stellar populations. Thanks to Gaia's astrometry, large spectroscopic surveys, and improved age-dating methods, this picture is changing. Aims. We aim to study spatial and kinematic substructures of the region encompassed by Upper Scorpius and Ophiuchus star forming regions. We want to determine dynamical traceback ages and study the star formation history of the complex. Methods. We combined our spectroscopic observations with spectra in public archives and large radial velocity surveys to obtain a precise radial velocity sample to complement the Gaia astrometry. We used a Gaussian Mixture Model to identify different kinematic structures in the 6D space of positions and velocities. We applied an orbital traceback analysis to estimate a dynamical traceback age for each group and determine the place where it was born. Results. We identified seven different groups in this region. Four groups (ν Sco, β Sco, σ Sco and δ Sco) are part of Upper Scorpius, two groups (ρ Oph and α Sco) are in Ophiuchus, and another group (π Sco) is a nearby young population. We found an age gradient from the ρ Oph group (the youngest) to the δ Sco group ( 5 Myr), showing that star formation was a sequential process for the past 5 Myr. Our traceback analysis shows that Upper Scorpius and ρ Oph groups share a common origin. The closer group of π Sco is probably older, and the traceback analysis suggests that this group and the α Sco group have a different origin, likely related to other associations in the Sco-Cen complex. Conclusions. Our study shows that this region has a complex star formation history that goes beyond the current formation scenario, likely a result of stellar feedback from massive stars, supernovae explosions, and dynamic interactions between stellar groups and the molecular gas. In particular, we speculate that photo-ionisation from the massive δ Sco star could have triggered star formation first in the β Sco group and then in the ν Sco group. The perturbations of stellar orbits due to stellar feedback and dynamical interactions could also be responsible for the 1-3 Myr difference that we found between dynamical traceback ages and isochronal ages.