The melanogenesis inhibition effect in zebrafish (Danio rerio) and antityrosinase activity of the ethanolic extract and its phytochemicals from Ceylon olive (Elaeocarpus serratus Linn.) leaves were investigated in this study. Among the leaf extract and four soluble fractions, the ethyl acetate soluble fraction exhibits the best antityrosinase and antimelanogenesis activities. One phenolic acid, gallic acid, and two flavonoids, myricetin and mearnsetin, are isolated from the active subfractions through the bioassay-guided isolation; their structures are elucidated based on the 1D and 2D NMR, FTIR, UV, and MS spectroscopic analyses. These compounds have significant antityrosinase activity whether using l-tyrosine or l-DOPA as the substrate; mearnsetin shows the optimal activity. In the enzyme kinetic investigation, both gallic acid and mearnsetin are the competitive-type inhibitors against mushroom tyrosinase, and myricetin acts as a mixed-type tyrosinase inhibitor. Leaf extract and an ethyl acetate soluble fraction show effective performance in the inhibition of melanin formation in zebrafish embryos. Mearnsetin also possesses a promising antimelanogenesis effect, which is superior to the positive control, arbutin. Results reveal that the Ceylon olive leaf extract and its phytochemicals, especially mearnsetin, have the potential to be used as antimelanogenesis and skin-whitening ingredients.