Salt stress is a complex trait that poses a grand challenge in developing new crops better adapted to saline environments. Some plants, called recretohalophytes, that have naturally evolved to secrete excess salts through salt glands, offer an underexplored genetic resource for examining how plant development, anatomy, and physiology integrate to prevent excess salt from building up to toxic levels in plant tissue. In this review we examine the structure and evolution of salt glands, salt gland-specific gene expression, and the possibility that all salt glands have originated via evolutionary modifications of trichomes. Salt secretion via salt glands is found in more than 50 species in 14 angiosperm families distributed in caryophyllales, asterids, rosids, and grasses. The salt glands of these distantly related clades can be grouped into four structural classes. Although salt glands appear to have originated independently at least 12 times, they share convergently evolved features that facilitate salt compartmentalization and excretion. We review the structural diversity and evolution of salt glands, major transporters and proteins associated with salt transport and secretion in halophytes, salt gland relevant gene expression regulation, and the prospect for using new genomic and transcriptomic tools in combination with information from model organisms to better understand how salt glands contribute to salt tolerance. Finally, we consider the prospects for using this knowledge to engineer salt glands to increase salt tolerance in model species, and ultimately in crops.