The methodology of mixture experiments has been applied to optimize the multicomponent solvent composition for high performance thin-layer chromatography (HPTLC). In this study, a mixture of five solvents for the mobile phase was maximized for the separation of twelve structurally related B(a)P metabolites on normal phase HPTLC plates. Volume fractions (rather than absolute quantities) of the mobile phase solvents were constrained by upper bounds by considering solvent strength. These restrictions defined an experimental region that is an irregular polyhedron formed from a truncated 4-dimensional simplex.The optimization of the expected separation response over the experimental region was based on a second-order Scheffk polynomial estimated from twenty-five experimental runs. Six functions for measuring separation based on R, values were evaluated. A new response function is proposed that overcomes the difficulties of previously published criteria.