Numerous natural and technological phenomena are governed by resonances. In nanophotonics, resonances often result from the interaction of several optical elements. Controlling these resonances is an excellent opportunity to provide light with properties on demand for applications ranging from sensing to quantum technologies. The inverse design of large, distributed resonators, however, is typically challenged by high computational costs when discretizing the entire system in space. Here, this limitation is overcome by harnessing prior knowledge about the individual scatterers that form the resonator and their interaction. In particular, a transition matrix multi‐scattering framework is coupled with the state‐of‐the‐art adaptive Antoulas–Anderson (AAA) algorithm to identify complex poles of the optical response function. A sample refinement strategy suitable for accurately locating a large number of poles is introduced. The AAA algorithm is tied into an automatic differentiation framework to efficiently differentiate multi‐scattering resonance calculations. The resulting resonance solver allows for efficient gradient‐based optimization, demonstrated here by the inverse design of an integrated exciton‐polariton cavity. This contribution serves as an important step towards efficient resonance calculations in a variety of multi‐scattering scenarios, such as inclusions in stratified media, periodic lattices, and scatterers with arbitrary shapes.