As transport properties of doped ceria electrolytes depend significantly on the nature of the dopant and the defectivity, the design of new materials and devices requires proper understanding of the defect structure. Among lanthanide dopants, Yb shows some peculiar characteristics that call for a possible different defect structure compared to Gd and Sm conventional dopants, which could be linked to its poorer performance. For this purpose, we combine synchrotron and neutron powder diffraction exploiting the Rietveld and Pair distribution Function. By increasing its concentration, Yb produces qualitatively the same structural distortions as other dopants, leading to a domain structure involving the progressive nucleation and growth of nanodomains with a Yb2O3-like (C-type) structure hosted in a fluorite CeO2 matrix. However, when it comes to growing the C-type nanodomains into a long-range phase, the transformation is less pronounced. At the same time, a stronger structural distortion occurs at the local scale, which is consistent with the segregation of a large amount of oxygen vacancies. The strong trapping of VOs by Yb3+ explains the poor performance of Yb-doped ceria with respect to conventional Sm-, Gd-, and Y-doped samples at equal temperature and dopant amount.