In recent years, many effective constrained multi-objective evolutionary algorithms (CMOEAs) have been proposed and successfully applied to address constrained multi-objective optimization problems (CMOPs). Nevertheless, few CMOEAs have fully explored CMOPs with imbalanced objectives or constraints. In this study, we propose a hybrid algorithm called M2M-IEpsilon to handle CMOPs with such characteristics, which combines an improved epsilon constraint-handling method (IEpsilon) with a multi-objective to multi-objective (M2M) decomposition strategy. The M2M decomposition mechanism divides a population into a set of sub-populations, which strengthens the diversity of the population. The IEpsilon constraint-handling method enables individuals with small constraint violation values to survive to the next generation, thus leading to a search for promising regions. In addition, a series of imbalanced CMOPs, named IM-CMOPs, is designed to verify the performance of the proposed M2M-IEpsilon algorithm. The comprehensive experimental results indicate that the proposed method can solve such imbalanced CMOPs well and perform significantly better than the other eight new or classical CMOEAs. Finally, we used the proposed M2M-IEpsilon to optimize the wellbore trajectory problem, and it achieved good results compared with previously developed algorithms.