Mangosteen peel, a waste produced during mangosteen processing, has been reported to be rich in xanthone and anthocyanin, both of which possess vital biological activities such as anti-cancer properties. The objectives of this study were to analyze various xanthones and anthocyanins in mangosteen peel by UPLC-MS/MS for the subsequent preparation of both xanthone and anthocyanin nanoemulsions to study their inhibition effects on liver cancer cells HepG2. Results showed that methanol was the optimal solvent for the extraction of xanthones and anthocyanins, with a total amount of 68,543.39 and 2909.57 μg/g, respectively. A total of seven xanthones, including garcinone C (513.06 μg/g), garcinone D (469.82 μg/g), γ-mangostin (11,100.72 μg/g), 8-desoxygartanin (1490.61 μg/g), gartanin (2398.96 μg/g), α-mangostin (51,062.21 μg/g) and β-mangostin (1508.01 μg/g), as well as two anthocyanins including cyanidin-3-sophoroside (2889.95 μg/g) and cyanidin-3-glucoside (19.72 μg/g), were present in mangosteen peel. The xanthone nanoemulsion was prepared by mixing an appropriate portion of soybean oil, CITREM, Tween 80 and deionized water, while the anthocyanin nanoemulsion composed of soybean oil, ethanol, PEG400, lecithin, Tween 80, glycerol and deionized water was prepared as well. The mean particle size of the xanthone extract and nanoemulsion were, respectively, 22.1 and 14.0 nm as determined by DLS, while the zeta potential was −87.7 and −61.5 mV. Comparatively, xanthone nanoemulsion was more effective than xanthone extract in inhibiting the growth of HepG2 cells, with the IC50 being 5.78 μg/mL for the former and 6.23 μg/mL for the latter. However, the anthocyanin nanoemulsion failed to inhibit growth of HepG2 cells. Cell cycle analysis revealed that the proportion of the sub-G1 phase followed a dose-dependent increase, while that of the G0/G1 phase showed a dose-dependent decline for both xanthone extracts and nanoemulsions, with the cell cycle being possibly arrested at the S phase. The proportion of late apoptosis cells also followed a dose-dependent rise for both xanthone extracts and nanoemulsions, with the latter resulting in a much higher proportion at the same dose. Similarly, the activities of caspase-3, caspase-8 and caspase-9 followed a dose-dependent increase for both xanthone extracts and nanoemulsions, with the latter exhibiting a higher activity at the same dose. Collectively, xanthone nanoemulsion was more effective than xanthone extract in inhibiting the growth of HepG2 cells. Further research is needed to study the anti-tumor effect in vivo.