Due to the volatility and uncertainty of the financial market, investors often use the form of portfolio to actively manage their assets. Portfolio optimization (PO) is becoming more and more important for investors. However, PO is frequently a kind of NP-hard problem in the field of modern financial optimization, which has gradually attracted the attention and interest of researchers. Some efficient mathematical models were built to describe the return and risk of portfolio. A lot of precise and approximate fast algorithms are used to solve the established PO models. The fundamental purpose is to maximize the return and to minimize the risk of portfolio under certain constraints. In recent years, researchers not only limit the goal of PO to the balance between risk and return, but also pay attention to liquidity, environmental, social, and governance (ESG) controversy level, Sortino ratio, and other indicators. The number of PO targets and constraints is further extended. In the past two decades, swarm intelligence (SI) algorithms have been widely introduced to solve PO problems. SI algorithm is mainly inspired from the daily phenomena in nature or self-organization, self-adaptation, and self-learning of biological population. The existing research results show that SI algorithm has the characteristics of high efficiency and can obtain satisfactory solutions in solving PO problems. The recent advances on the classic portfolio optimization concepts, models, and the usual SI-based solving algorithms are presented. Finally, future potential research directions are presented.