The dense matter equation of state (EOS) determines neutron star (NS) structure but can be calculated reliably only up to one to two times the nuclear saturation density, using accurate manybody methods that employ nuclear interactions from chiral effective field theory constrained by scattering data. In this work, we use physically motivated ansatzes for the speed of sound c S at high density to extend microscopic calculations of neutron-rich matter to the highest densities encountered in stable NS cores. We show how existing and expected astrophysical constraints on NS masses and radii from X-ray observations can constrain the speed of sound in the NS core. We confirm earlier expectations that c S is likely to violate the conformal limit of c 2 S ≤ c 2 /3, possibly reaching values closer to the speed of light c at a few times the nuclear saturation density, independent of the nuclear Hamiltonian. If QCD obeys the conformal limit, we conclude that the rapid increase of c S required to accommodate a 2 M NS suggests a form of strongly interacting matter where a description in terms of nucleons will be unwieldy, even between one and two times the nuclear saturation density. For typical NSs with masses in the range 1.2 − 1.4 M , we find radii between 10 and 14 km, and the smallest possible radius of a 1.4 M NS consistent with constraints from nuclear physics and observations is 8.4 km. We also discuss how future observations could constrain the EOS and guide theoretical developments in nuclear physics.