Point intrinsic and extrinsic defects, especially paramagnetic ions of transition metals and rare-earth elements, have essential influence on properties of lithium niobate, LN and tantalate, LT, and often determine their suitability for numerous applications. Discussions about structures of the defects in LN/LT have lasted for decades. Many experimental methods facilitate progress in determining the structures of impurity centers. This paper gives current bird’s eye view on contributions of Electron Paramagnetic Resonance (EPR), and Electron Nuclear Double Resonance (ENDOR) studies to the determination of impurity defect structures in LN and LT crystals for a broad audience of researchers and students. Symmetry and charge compensation considerations restrict a number of possible structures. Comparison of measured angular dependences of ENDOR frequencies with calculated ones for Li and Nb substitution using dipole–dipole approximation allows unambiguously to determine the exact location of paramagnetic impurities. Models with two lithium vacancies explain angular dependencies of EPR spectra for Me3+ ions substituting for Li+ like Cr, Er, Fe, Gd, Nd, and Yb. Self-compensation of excessive charges through equalization of concentrations of Me3+(Li+) and Me3+(Nb5+) and appearance of interstitial Li+ in the structural vacancy near Me3+(Nb5+) take place in stoichiometric LN/LT due to lack of intrinsic defects.