The problem of portfolio management relates to the selection of optimal stocks, which results in a maximum return to the investor while minimizing the loss. Traditional approaches usually model the portfolio selection as a convex optimization problem and require the calculation of gradient. Note that gradient-based methods can stuck at local optimum for complex problems and the simplification of portfolio optimization to convex, and further solved using gradient-based methods, is at a high cost of solution accuracy. In this paper, we formulate a nonconvex model for the portfolio selection problem, which considers the transaction cost and cardinality constraint, thus better reflecting the decisive factor affecting the selection of portfolio in the real-world. Additionally, constraints are put into the objective function as penalty terms to enforce the restriction. Note that this reformulated problem cannot be readily solved by traditional methods based on gradient search due to its nonconvexity. Then, we apply the Beetle Antennae Search (BAS), a nature-inspired metaheuristic optimization algorithm capable of efficient global optimization, to solve the problem. We used a large real-world dataset containing historical stock prices to demonstrate the efficiency of the proposed algorithm in practical scenarios. Extensive experimental results are presented to further demonstrate the efficacy and scalability of the BAS algorithm. The comparative results are also performed using Particle Swarm Optimizer (PSO), Genetic Algorithm (GA), Pattern Search (PS), and gradient-based fmincon (interior-point search) as benchmarks. The comparison results show that the BAS algorithm is six times faster in the worst case (25 times in the best case) as compared to the rival algorithms while achieving the same level of performance. INDEX TERMS Portfolio management, constrained optimization, nature-inspired algorithms, beetle search optimization.