We present a comprehensive study on how to design and fabricate low loss electro-optic phase shifters based on an electro-optic polymer and the silicon nitride and silicon oxynitride waveguide material systems. The loss mechanisms of phase shifters with an electro-optic (EO) polymer cladding are analyzed in detail and design solutions to achieve lowest losses are presented. In order to verify the low loss design a proof of concept prototype phase shifter was fabricated, which exhibits an attenuation of 0.8 dB/cm at 1550 nm and an electro-optic efficiency factor of 27%. Furthermore, the potential of this class of phase shifters is evaluated in numerical simulations, from which the optimal design parameters and achievable figures of merit were derived. The presented phase shifter design has its potential for application in fast adaptive multi stage devices for optical signal processing.