Androdioecy is the crucial transition state in the evolutionary direction of hermaphroditism to dioecy, however, the molecular mechanisms underlying the formation of this sex system remain unclear. While popular in China for its ornamental and cultural value, Osmanthus fragrans has an extremely rare androdioecy breeding system, meaning that there are both male and hermaphroditic plants in a population. To unravel the mechanisms underlying the formation of androdioecy, we performed small RNA sequencing studies on male and hermaphroditic O. fragrans. A total of 334 miRNAs were identified, of which 59 were differentially expressed. Functional categorization revealed that the target genes of differentially expressed miRNAs were mainly involved in the biological processes of reproductive development and the hormone signal transduction pathway. We speculated that the miRNA160, miRNA167, miRNA393 and miRNA396 families may influence the sex differentiation in O. fragrans. Overall, our study is the first exploration of miRNAs in the growth and development process of O. fragrans, and is also the first study of androdioecious plants from the miRNA sequencing perspective. The analysis of miRNAs and target genes that may be involved in the sex differentiation process lay a foundation for the ultimate discovery of the androdioecious molecular mechanism in O. fragrans.