Invaginating structures are common in the synapses of most animals. However, the details of these invaginating structures remain understudied in part because they are not well resolved in light microscopy and were often misidentified in early electron microscope (EM) studies. Utilizing experimental techniques along with the latest advances in microscopy, such as focused ion beam-scanning EM (FIB-SEM), evidence is gradually building to suggest that the synaptic invaginating structures contribute to synapse development, maintenance, and plasticity. These invaginating structures are most elaborate in synapses mediating rapid integration of signals, such as muscle contraction, mechanoreception, and vision. Here we argue that the synaptic invaginations should be considered in future studies seeking to understand their role in sensory integration and coordination, learning, and memory. We review the various types of invaginating structures in the synapses and discuss their potential functions. We also present several new examples of invaginating structures from a variety of animals including Drosophila and mice, mainly using FIB-SEM, with which we trace the form and arrangement of these structures.