The basic idea behind Rayleigh's criterion on resolving two incoherent optical point sources is that the overlap between the spatial modes from different sources would reduce the estimation precision for the locations of the sources, dubbed Rayleigh's curse. We generalize the concept of Rayleigh's curse to the abstract problems of quantum parameter estimation with incoherent sources. To manifest the effect of Rayleigh's curse on quantum parameter estimation, we define the curse matrix in terms of quantum Fisher information and introduce the global and local immunity to the curse accordingly. We further derive the expression for the curse matrix and give the necessary and sufficient condition on the immunity to Rayleigh's curse. For estimating the one-dimensional location parameters with a common initial state, we demonstrate that the global immunity to the curse on quantum Fisher information is impossible for more than two sources.