We present a formulation for the pole expansion of the scattering matrix of open optical resonators, in which the pole contributions are expressed solely in terms of the resonant states, their wave numbers, and their electromagnetic fields. Particularly, our approach provides an accurate description of the optical scattering matrix without the requirement of a fit for the pole contributions, or the restriction to geometries, or systems with low Ohmic losses. Hence, it is possible to derive the analytic dependence of the scattering matrix on the wave number with low computational effort, which allows for avoiding the artificial frequency discretization of conventional frequency-domain solvers of Maxwell's equations and for finding the optical far-and near-field response based on the physically meaningful resonant states. This is demonstrated for three test systems, including a chiral arrangement of nanoantennas, for which we calculate the absorption and the circular dichroism.