Background Cystic echinococcosis caused by the cestode Echinococcus granulosus remains a serious helminthic zoonosis affecting humans and animals in many endemic developing countries. Surgical intervention is the best management choice, although it is associated with high recurrence rates and serious complications. Also, the commonly used chemotherapeutics exhibited serious side effects. This study aimed to evaluate the protoscolicidal effects of eugenol (Eug) essential oil and its nanoemulsion (Eug-NE) against protoscoleces (PCs) of hydatid cysts in vitro. Methods Eug-NE was prepared and characterized. Their cytotoxicity on macrophages was assessed by the 3-(4,5-dimethylthiazol-2-yl)-2,5-diphenyltetrazolium bromide assay. E. granulosus PCs were treated with various concentrations of Eug and Eug-NE at different exposure times. The viability of protoscoleces was evaluated by the eosin exclusion test, and the changes in the morphology of protoscoleces were assessed. Albendazole (ABZ) was used as a positive control. Results The cellular cytotoxicity of Eug and Eug-NE on macrophage cells, in minimum and maximum concentrations (0.2 and 1 μl/mL), were nearly negligible ranging from 4.7% to 8.3% and 3.7% to 7.2%, respectively. The results showed highly significant activity of Eug-NE and Eug against hydatid PCs compared to ABZ (P < 0.05). Eug and Eug-NE have similar protoscolicidal effects at all used concentrations. Their highest scolicidal activity (100% mortality rate) was recorded at 1 μl/ml after 30 min incubation (LC50 = 0.298—LC90 = 0.521 and LC50 = 0.309—LC90 = 0.646, respectively). Both formulations showed time- and dose-dependent effects. Conclusions This study suggested the potent scolicidal activities of Eug and Eug-NE as promising alternative scolicidal agents. Future studies are recommended to explore the mechanism of action and treatment response in vivo and clinical settings.