A bipartite system is defined as two microscopic entities being able to exchange energy. When excited by light, the complete optical response functions at first (polarizabilities) and second orders (first hyperpolarizabilities) of such a system are determined using the diagrammatic theory of optics. The generality of the method is ensured by the free choice of light–matter and matter–matter interaction Hamiltonians and by the arbitrary number of quanta involved in the energy exchange. In the dipolar approximation, the optical response functions of the system (i.e., of the interacting entities) are linked to the responses of the interaction-free entities by transfer matrices. These universal matrices identically modify the optical response functions at all orders in the electromagnetic field, allowing the implementation of matter–matter interactions in higher-order processes, such as stimulated or spontaneous Raman scattering and four-wave mixing. This formalism is then applied to various composite systems: dimers, multimers and lattices of nanoparticles and molecules, dense molecular layers, and substrate-induced image dipoles.