We describe a method for performing cryogenic permittivity and loss tangent measurements of small (few mm), high-permittivity dielectric cylinders used in surface impedance measurement of superconductors. We combine the use of sapphire and PTFE supports to hold the dielectric under test and provide good thermal conductivity to the cold head. Additionally, the sapphire support is used to facilitate the measurement of the cavity's metal loss and the assessment of its contribution to the overall resonator loss.