Fluorescent nanodiamonds (NDs) are strong contenders as bio-labels for life science imaging, diagnostics and therapeutics. Ultimately, for their use in biomedical applications, their size should ideally be less than 10 nm. Yet, even more critical for their specificity and efficient uptake in cellular systems, is their resilience to aggregation, which is dictated by their colloidal stability in complex, physiological environments. To this end, we characterize ultrasmall detonation NDs (~5 nm) by examining their surface chemical profiles and stability in solutions of varying ionic strength and pH. Using dynamic light scattering measurements, we demonstrate that ultra-small ND particles with chemically homogeneous and negatively charged surface profiles are more stable than positive particles under a broad range of simulated biological environments. We show that the colloidal stability of both positive and negative ultra-small NDs is improved by functionalization with bovine serum albumin. Based on these analyses, we propose and describe strategies for enhancing the overall colloidal stability of ultra-small NDs and their resilience to aggregation. Our findings provide a practical framework towards the reduction in size of the bio-conjugates employed to probe complex biological systems, and the advancement of bio-imaging techniques with minimal perturbation of the molecular trafficking in cellular and organelle systems.