Primordial magnetic fields are generated during inflation by considering actions that break the conformal invariance of the electromagnetic field. To break the conformal invariance, the electromagnetic fields are coupled either to the inflaton or to the scalar curvature. Also, a parity violating term is often added to the action in order to enhance the amplitudes of the primordial electromagnetic fields. In this work, we examine the effects of deviations from slow roll inflation on the spectra of non-helical as well as helical electromagnetic fields. We find that, in the case of the coupling to the scalar curvature, there arise certain challenges in generating electromagnetic fields of the desired shapes and strengths even in slow roll inflation. When the field is coupled to the inflaton, it is possible to construct model-dependent coupling functions which lead to nearly scale invariant magnetic fields in slow roll inflation. However, we show that sharp features in the scalar power spectrum generated due to departures from slow roll inflation inevitably lead to strong features in the power spectra of the electromagnetic fields. Moreover, we find that such effects can also considerably suppress the strengths of the generated electromagnetic fields over the scales of cosmological interest. We illustrate these aspects with the aid of specific inflationary models that have been considered to produce specific features in the scalar power spectrum. Further, we find that, in such situations, if the strong features in the electromagnetic power spectra are to be undone, the choice of the coupling function requires considerable fine tuning. We discuss wider implications of the results we obtain.