PurposeThe objective of the present work is to present the design optimization of composite cylindrical shell subjected to an axial compressive load and lateral pressure.Design/methodology/approachA novel optimization method is developed to predict the optimal fiber orientation in composite cylindrical shell. The optimization is carried out by coupling analytical and finite element (FE) results with a genetic algorithm (GA)-based optimization scheme developed in MATLAB. Linear eigenvalue were performed to evaluate the buckling behaviour of composite cylinders. In analytical part, besides the buckling analysis, Tsai-Wu failure criteria are employed to analyse the failure of the composite structure.FindingsThe optimal result obtained through this study is compared with traditionally used laminates with 0, 90, ±45 orientation. The results suggest that the application of this novel optimization algorithm leads to an increase of 94% in buckling strength.Originality/valueThe proposed optimal fiber orientation can provide a practical and efficient way for the designers to evaluate the buckling pressure of the composite shells in the design stage.