The performance of constrained multi-objective differential evolution algorithms (CMOEAs) is mainly determined by constraint handling techniques (CHTs) and their generation strategies. To realize the adaptive adjustment of CHTs and generation strategies, an adaptive constrained multi-objective differential evolution algorithm based on the state–action–reward–state–action (SARSA) approach (ACMODE) is introduced in the current study. In the proposed algorithm, the suitable CHT and the appropriate generation strategy can be automatically selected via a SARSA method. The performance of the proposed algorithm is compared with four other famous CMOEAs on five test suites. Experimental results show that the overall performance of the ACMODE is the best among all competitors, and the proposed algorithm is capable of selecting an appropriate CHT and a suitable generation strategy to solve a particular type of constrained multi-objective optimization problems.