In this research, the lateral buckling analysis and layup optimization of the laminated composite of web and flanges tapered thin-walled I-beams based on maximizing lateral-torsional stability strength and minimizing mass/cost of the structure are investigated. The classical lamination theory and Vlasov’s model for thin-walled cross-section are adopted to establish the total potential energy for thin-walled symmetric balanced laminated beams with varying I-section. By implementing the Ritz method, an explicit formulation for the lateral-torsional buckling load of a double-tapered beam subjected to transverse loading is then derived in terms of the load height parameter and stiffness quantities. Subsequently, the optimal arrangements of layer sequences are obtained using the non-dominated sorting genetic algorithm (NSGA-II) and properly defined objective function. The critical factors of fitness function as lateral buckling strength and the mass of the structure with critical limitations as ply angle, number of layers for the web and flanges, and the thickness of all section walls are considered in this study. Finally, the optimal layer arrangement for the web and flanges are separately determined and discussed. The results show that the presented optimization procedure and layups patterns lead to increasing the lateral-torsional buckling capacity about 52% compared to the conventional angle-ply and unidirectional layups for the web and flanges, respectively.