Viscoelastic flows are pervasive in a host of natural and industrial processes, where the emergence of nonlinear and time-dependent dynamics regulate flow resistance, energy consumption, and particulate dispersal. Polymeric stress induced by the advection and stretching of suspended polymers feeds back on the underlying fluid flow, which ultimately dictates the dynamics, instability, and transport properties of viscoelastic fluids. However, direct experimental quantification of the stress field is challenging, and a fundamental understanding of how Lagrangian flow structure regulates the distribution of polymeric stress is lacking. In this work, we show that the topology of the polymeric stress field precisely mirrors the Lagrangian stretching field, where the latter depends solely on flow kinematics. We develop a general analytical expression that directly relates the polymeric stress and stretching in weakly viscoelastic fluids for both nonlinear and unsteady flows, which is also extended to special cases characterized by strong kinematics. Furthermore, numerical simulations reveal a clear correlation between the stress and stretching field topologies for unstable viscoelastic flows across a broad range of geometries. Ultimately, our results establish a fundamental connection between the Eulerian stress field and the Lagrangian structure of viscoelastic flows, unifying these two disparate fields of continuum analysis. This work provides a simple framework to determine the topology of polymeric stress directly from readily measurable flow field data and lays the foundation for directly linking the polymeric stress to flow transport properties.