Context. The extragalactic distance scale builds on the Cepheid period-luminosity (PL) relation. Decades of work have not yet convincingly established the sensitivity of the PL relation to metallicity. This currently prevents a determination of the Hubble constant accurate to 1% from the classical Cepheid-SN Ia method. Aims. In this paper, we want to carry out a strictly differential comparison of the absolute PL relations obeyed by classical Cepheids in the Milky Way (MW), LMC and SMC galaxies. Taking advantage of the substantial metallicity difference among the Cepheid populations in these three galaxies, we want to establish a possible systematic trend of the PL relation absolute zero point as a function of metallicity, and determine the size of such an effect in optical and near-infrared photometric bands. Methods. We are using the IRSB Baade-Wesselink type method as calibrated by Storm et al. to determine individual distances to the Cepheids in our samples in MW, LMC and SMC. For our analysis, we use a greatly enhanced sample of Cepheids in the SMC (31 stars) as compared to the small sample (5 stars) available in our previous work. We use the distances to determine absolute Cepheid PL relations in optical and near-infrared bands in each of the three galaxies. Results. Our distance analysis of 31 SMC Cepheids with periods from 4-69 days yields tight PL relations in all studied bands, with slopes consistent with the corresponding LMC and MW relations. Adopting the very accurately determined LMC slopes for the optical and near-infrared bands, we determine the zero point offsets between the corresponding absolute PL relations in the 3 galaxies. We find that in all bands the metal-poor SMC Cepheids are intrinsically fainter than their more metal-rich counterparts in the LMC and MW. In the K band the metallicity effect is −0.23 ± 0.06 mag/dex while in the V, (V − I) Wesenheit index it is slightly stronger, −0.34±0.06 mag/dex. We find some evidence that the PL relation zero point-metallicity relation might be nonlinear, becoming steeper for lower metallicities. Conclusions. Using sizeable Cepheid samples in the MW, LMC and SMC with very accurate photometric and radial velocity data we establish the metallicity sensitivity of the Cepheid PL relations in the optical and near-infrared regimes. We find a significant effect in all bands in the sense that the more metal-poor SMC Cepheids are intrinsically fainter than their LMC and Galactic counterparts. We find suggestive evidence that the metallicity sensitivity of the PL relation might be nonlinear, being small in the range between solar and LMC Cepheid metallicity, and becoming steeper towards the lower-metallicity regime.