The theory is developed for one and two atom interactions when the atom has a Rydberg electron attached to a hyperfine split core state. This situation is relevant for some of the rare earth and alkaline earth atoms that have been proposed for experiments on Rydberg-Rydberg interactions. For the rare earth atoms, the core electrons can have a very substantial total angular momentum, J, and a non-zero nuclear spin, I. In the alkaline earth atoms there is a single, s, core electron whose spin can couple to a non-zero nuclear spin for odd isotopes. The resulting hyperfine splitting of the core state can lead to substantial mixing between the Rydberg series attached to different thresholds. Compared to the unperturbed Rydberg series of the alkali atoms, the series perturbations and near degeneracies from the different parity states could lead to qualitatively different behavior for single atom Rydberg properties (polarizability, Zeeman mixing and splitting, etc) as well as Rydberg-Rydberg interactions (C5 and C6 matrices).