Background Topical drug delivery using nanoemulsions and nanogels is a promising approach to treating skin disorders such as melanoma. Methods In this study, the chemical composition of Mentha pulegium essential oil with five major compounds, including pulegone (68.11%), l-menthone (8.83%), limonene (2.90%), iso-pulegone (2.69%), and iso-menthone (1.48%) was first identified using GC-MS (Gas chromatography–Mass Spectrometry) analysis. Afterward, a nano-scaled emulsion containing the essential oil with a droplet size of 7.70 ± 1 nm was prepared. Nanogel containing the essential oil was then prepared by adding (2% w/v) carboxymethyl cellulose to the nano-scaled emulsion. Moreover, the successful loading of M. pulegium essential oil in the nano-scaled emulsion and nanogel was confirmed using ATR-FTIR (Attenuated total reflectance-Fourier Transform InfraRed) analysis. Then, human A375 melanoma cells were treated with different concentrations of samples, the MTT assay evaluated cell viability, and cell apoptosis was confirmed by flow cytometry. In addition, the expression of apoptotic and anti-apoptotic genes, including Bax and Bcl-2, was evaluated using the qPCR (quantitative Polymerase Chain Reaction) technique. Results The results showed that cell viability was reduced by 90 and 45% after treatment with 300 μg/mL of the nanogel and nano-scaled emulsion. As confirmed by flow cytometry, this effect was mediated by apoptosis. Furthermore, gene expression analysis showed up-regulation of Bax and down-regulation of Bcl-2 genes. Therefore, the prepared nanogel, with high efficacy, could be considered a potent anticancer agent for supplementary medicine and in vivo research.