Most of the free parameters in the Standard Model (SM) -a quantum field theory which has successfully elucidated the behaviors of strong, weak and electromagnetic interactions of all the known fundamental particles, come from the lepton and quark flavors. The discovery of neutrino oscillations has proved that the SM is incomplete, at least in its lepton sector; and thus the door of opportunity is opened to exploring new physics beyond the SM and solving a number of flavor puzzles. In this review article we give an overview of important progress made in understanding the mass spectra, flavor mixing patterns, CP-violating effects and underlying flavor structures of charged leptons, neutrinos and quarks in the past twenty years. After introducing the standard pictures of fermion mass generation, flavor mixing and CP violation in the SM extended with the presence of massive Dirac or Majorana neutrinos, we briefly summarize current experimental knowledge about the flavor parameters of quarks and leptons. Various ways of describing flavor mixing and CP violation are discussed, the renormalization-group evolution of flavor parameters is illuminated, and the matter effects on neutrino oscillations are interpreted. Taking account of possible extra neutrino species, we propose a standard parametrization of the 6 × 6 flavor mixing matrix and comment on the phenomenological aspects of heavy, keV-scale and light sterile neutrinos. We pay particular attention to those novel and essentially model-independent ideas or approaches regarding how to determine the Yukawa textures of Dirac fermions and the effective mass matrix of Majorana neutrinos, including simple discrete and continuous flavor symmetries. An outlook to the future development in unravelling the mysteries of flavor structures is also given.