Silica aerogels are known to be materials with exceptional characteristics, such as ultra-low density, high surface area, high porosity, high adsorption, and low-thermal conductivity. In addition, these unique properties are mainly related to their specific processing. Depending on the aerogel synthesis procedure, the aerogels texture can be tailored with meso and/or macroporosity. Fractal geometry has been observed and used to describe silica aerogels at nanoscales in certain conditions. In this review paper, we describe the fractal structure of silica aerogels that can develop depending on the synthesis conditions. X-ray and neutron scattering measurements allow to show that silica aerogels can exhibit a fractal structure over one or even more than two orders of magnitude in length. The fractal dimension does not depend directly on the material density but can vary with the synthesis conditions. It ranges typically between 1.6 and 2.4. The effect of the introduction of silica particles or of further thermal treatment or compression of the silica aerogels on their microstructure and their fractal characteristics is also resumed.