We show how to construct a linearly independent set of antisymmetrized geminal power (AGP) states, which allows us to rewrite our recently introduced geminal replacement (GR) models as linear combinations of non-orthogonal AGPs. This greatly simplifies the evaluation of matrix elements and permits us to introduce an AGP-based selective configuration interaction (SCI) method, which can reach arbitrary excitation levels relative to a reference AGP, balancing accuracy and cost as we see fit.