The elastic modulus of colloidal fat crystal networks scales with the volume fraction of solids in a power-law fashion. To explain and predict how the elastic properties of these networks change with their volume fraction of solids, several physical models have been proposed. In this review, the chronology of the development of structural-mechanical models to explain the elasticity of fats is reviewed, leading to the development of the fractal model. In the fractal model, the fractal-like behavior of fat crystal networks, which can be considered fractal gels of polycrystals in oil, or colloidal crystals, is used to explain the power-law scaling behavior of the shear elastic modulus to the volume fraction of solids. Lately, however, many experimental results and simulation studies suggest that the stress distribution within networks can be dramatically heterogeneous, which means that a small part of the network carries most of the stress. This concept was introduced into a modified fractal model by deriving an expression for the effective volume fraction of stress-carrying solids. The modified fractal model fits the experimental data well and successfully explains the sometimes observed non-linear log-log behavior between the shear elastic modulus and the volume fraction of solids.