Describing non-equilibrium properties of quantum many-body systems is challenging due to high entanglement in the wavefunction. We take an open-quantum-system viewpoint and describe evolution of local observables in terms of the influence matrix (IM), which encodes the effects of a many-body system as an environment for a local subsystem. Recent works found that in many dynamical regimes the IM of an infinite system has low temporal entanglement and can be efficiently represented as a matrix-product state (MPS). Yet, direct iterative constructions of the IM encounter highly entangled intermediate states -a temporal entanglement barrier (TEB). We argue that TEB is ubiquitous, and elucidate its physical origin via a semiclassical quasiparticle picture that captures the exact behavior of integrable spin chains. Further, we show that a TEB also arises in chaotic spin chains, which lack well-defined quasiparticles. Based on these insights, we formulate an alternative light-cone growth algorithm, which provably avoids TEB, thus providing an efficient construction of the thermodynamic-limit IM as a MPS. This work demonstrates the efficiency of the IM approach to studying thermalization and transport in strongly interacting quantum systems.