A perturbation theory of the static response of insulating crystals to homogeneous electric fields, that combines the modern theory of polarization with the variation-perturbation framework is developed, at unrestricted order of perturbation. Conceptual issues involved in the definition of such a perturbative approach are addressed. In particular, we argue that the position operator, x, can be substituted by the derivative with respect to the wavevector, ∂/∂k, in the definition of an electric-field-dependent energy functional for periodic systems. Moreover, due to the unbound nature of the perturbation, a regularization of the Berry-phase expression for the polarization is needed in order to define a numerically-stable variational procedure. Regularization is achieved by means of a discretization procedure, which can be performed either before or after the perturbation expansion. We compare the two possibilities, show that they are both valid, and analyze their behavior when applied to a model tight-binding Hamiltonian. Lowest-order as well as generic formulas are presented for the derivatives of the total energy, the normalization condition, the 1 eigenequation, and the Lagrange parameters.