Many skin-whitening compounds target tyrosinase because it catalyzes two rate-limiting steps in melanin synthesis. Although many tyrosinase inhibitors are currently available for a skin–whitening purpose, undesirable adverse effects are also reported. Thus, numerous efforts have been made to develop safer tyrosinase inhibitors from natural products. In line with this, we tested fifty flavonoids, a group of naturally occurring antioxidants and metal chelators, and screened swertiajaponin as the strongest tyrosinase inhibitor in cell-free experiments. Swertiajaponin did not show cytotoxicity in B16F10, HaCat, and Hs27 cells and exhibited strong anti oxidative activity in experiments using the cell-free system and B16F10 cells. It markedly inhibited αMSH- or UVB-induced melanin accumulation in B16F10 cells and suppressed skin pigmentation in a human skin model. As underlying mechanisms, in silico and Lineweaver-Burk plot analyses exhibited that swertiajaponin may directly bind to and inhibit tyrosinase activity by forming multiple hydrogen bonds and hydrophobic interactions with the binding pocket of tyrosinase. In addition, western blotting results indicated that swertiajaponin inhibited oxidative stress-mediated MAPK/MITF signaling, leading to decrease in tyrosinase protein level. Together, swertiajaponin suppresses melanin accumulation by inhibiting both activity and protein expression levels of tyrosinase. Thus, it would be a novel additive for whitening cosmetics.