The neutron lifetime anomaly has been used to motivate the introduction of new physics with hidden-sector particles coupled to baryon number, and on which neutron stars provide powerful constraints. Although the neutron lifetime anomaly may eventually prove to be of mundane origin, we use it as motivation for a broader review of the ways that baryon number violation, be it real or apparent, and dark sectors can intertwine and how neutron star observables, both present and future, can constrain them.