We construct rest‐frame luminosity functions (LFs) at 3.6, 4.5, 5.8, 8 and 24 μm over the redshift range 0 < z < 2 for galaxies and 0 < z < 4 for optical quasi‐stellar objects (QSOs), using optical and infrared (IR) data from the Spitzer Wide‐area Infrared Extragalactic (SWIRE) Survey. The 3.6‐ and 4.5‐μm galaxy LFs show evidence for moderate positive luminosity evolution up to z∼ 1.5, consistent with the passive ageing of evolved stellar populations. Their comoving luminosity density was found to evolve passively, gradually increasing out to z∼ 0.5–1 but flattening, or even declining, at higher redshift. Conversely, the 24‐μm galaxy LF, which is more sensitive to obscured star formation and/or active galactic nuclei (AGN) activity, undergoes strong positive evolution, with the derived IR energy density and star formation rate (SFR) density ∝ (1 +z)γ with γ= 4.5+0.7−0.6 and the majority of this evolution occurring since z∼ 1. Optical QSOs, however, show positive luminosity evolution in all bands, out to the highest redshifts (3 < z < 4). Modelling as L*∝ (1 +z)γ gave γ= 1.3+0.1−0.1 at 3.6 μm, γ= 1.0+0.1−0.1 at 4.5 μm and stronger evolution at the longer wavelengths (5.8, 8 and 24 μm), of γ∼ 3. Comparison of the galaxy LFs to predictions from a semi‐analytic model based on cold dark matter (CDM) indicates that an initial mass function (IMF) skewed towards higher mass star formation in bursts compared to locally be preferred. As a result, the currently inferred massive SFRs in distant submm sources may require substantial downwards revision.