As it is becoming extremely competitive in software industry, large software companies have to select their project portfolio to gain maximum return with limited resources under many constraints. Project portfolio optimization using multiobjective evolutionary algorithms is promising because they can provide solutions on the Pareto-optimal front that are difficult to be obtained by manual approaches. In this paper, we propose an improved MOEA/D (multiobjective evolutionary algorithm based on decomposition) based on reference distance (MOEA/D_RD) to solve the software project portfolio optimization problems with optimizing 2, 3, and 4 objectives. MOEA/D_RD replaces solutions based on reference distance during evolution process. Experimental comparison and analysis are performed among MOEA/D_RD and several state-of-the-art multiobjective evolutionary algorithms, that is, MOEA/D, nondominated sorting genetic algorithm II (NSGA2), and nondominated sorting genetic algorithm III (NSGA3). The results show that MOEA/D_RD and NSGA2 can solve the software project portfolio optimization problem more effectively. For 4-objective optimization problem, MOEA/D_RD is the most efficient algorithm compared with MOEA/D, NSGA2, and NSGA3 in terms of coverage, distribution, and stability of solutions.