For spacecraft swarms, the multi-agent localization algorithm must scale well with the number of spacecraft and adapt to time-varying communication and relative sensing networks. In this paper, we present a decentralized, scalable algorithm for swarm localization, called the Decentralized Pose Estimation (DPE) algorithm. The DPE considers both communication and relative sensing graphs and defines an observable local formation. Each spacecraft jointly localizes its local subset of spacecraft using direct and communicated measurements. Since the algorithm is local, the algorithm complexity does not grow with the number of spacecraft in the swarm. As part of the DPE, we present the Swarm Reference Frame Estimation (SRFE) algorithm, a distributed consensus algorithm to coestimate a common Local-Vertical, Local-Horizontal (LVLH) frame. The DPE combined with the SRFE provides a scalable, fully-decentralized navigation solution that can be used for swarm control and motion planning. Numerical simulations and experiments using Caltech's robotic spacecraft simulators are presented to validate the effectiveness and scalability of the DPE algorithm.