The effective method of transformed differential quadrature (TDQ) is improved for the buckling design optimization of functionally graded (FG) carbon nanotube‐reinforced laminated composite quadrilateral (FG‐CNTRCQ) plates with arbitrary straight‐sided based on the first‐order shear deformation theory. The laminated plates with both uniformly and FG distributions of single‐walled carbon nanotubes varied along the thickness of layers are considered. The TDQ method is used to directly discretize the governing differential equations for an arbitrary straight‐sided physical domain without any need to transform the equations to the computational domain. In the presented genetic algorithm, the orientation of fibers is considered as the design parameter to obtain the optimum critical buckling loads in terms of different parameters including distributions of the fibers, number of layers, stacking sequence, CNTs volume fraction, geometrical parameters, and boundary conditions. The edges of the plates can take arbitrary end supports of simply and clamped boundary conditions. The formulation is developed based on the invariant terms to provide a systematic procedure with minimum computational cost. Some parametric studies are conducted to examine the effectiveness of the model for the design purpose of the FG‐CNTRCQ plates. POLYM. COMPOS., 39:E853–E868, 2018. © 2017 Society of Plastics Engineers