To take advantage of the singular properties of matter, as well as to characterize it, we need to interact with it. The role of optical spectroscopies is to enable us to demonstrate the existence of physical objects by observing their response to light excitation. The ability of spectroscopy to reveal the structure and properties of matter then relies on mathematical functions called optical (or dielectric) response functions. Technically, these are tensor Green’s functions, and not scalar functions. The complexity of this tensor formalism sometimes leads to confusion within some articles and books. Here, we do clarify this formalism by introducing the physical foundations of linear and non-linear spectroscopies as simple and rigorous as possible. We dwell on both the mathematical and experimental aspects, examining extinction, infrared, Raman and sum-frequency generation spectroscopies. In this review, we thus give a personal presentation with the aim of offering the reader a coherent vision of linear and non-linear optics, and to remove the ambiguities that we have encountered in reference books and articles.