We construct a physics-agnostic approach based on the sound speed model, which connects both low-density information from nuclear theory and high-density constraints from perturbative QCD (pQCD). Using this approach, we study the impact of pQCD constraints on neutron star (NS) equations of state (EoS) that have been constrained by astrophysical observations. We find that the pQCD constraints affect the EoS mainly beyond the densities realized in NS. Furthermore, we observe an interesting interplay between pQCD and astrophysical constraints, with pQCD preferring softer EoS for the heaviest NS while recent NICER observations suggest the EoS to be stiffer. We explore the sensitivity of our findings to pQCD uncertainties and we study the constraining power of pQCD if future observations of heavy NS were to suggest radii larger than 13 km for heavy stars.