Flavor symmetric model is one of the attractive Beyond Standard Models (BSMs) to reveal the flavor structure of the Standard Model (SM). A lot of efforts have been put into the model building and we find many kinds of flavor symmetries and setups are able to explain the observed fermion mass matrices. In this paper, we look for common predictions of physical observables among the ones in flavor symmetric models, and try to understand how to test flavor symmetry in experiments. Especially, we focus on the BSMs for leptons with extra Higgs SU(2) L doublets charged under flavor symmetry. In many flavor models for leptons, remnant symmetry is partially respected after the flavor symmetry breaking, and it controls well the Flavor Changing Neutral Currents (FCNCs) and suggests some crucial predictions against the flavor changing process, although the remnant symmetry is not respected in the full lagrangian. In fact, we see that τ − → e + µ − µ − (µ + e − e − ) and e + e − → τ + τ − (µ − µ + ) processes are the most important in the flavor models that the extra Higgs doublets belong to triplet representation of flavor symmetry. For instance, the stringent constraint from the µ → eγ process could be evaded according to the partial remnant symmetry. We also investigate the breaking effect of the remnant symmetry mediated by the Higgs scalars, and investigate the constraints from the flavor physics: the flavor violating τ and µ decays, the electric dipole moments, and the muon anomalous magnetic moment. We also discuss the correlation between FCNCs and nonzero θ 13 , and point out the physical observables in the charged lepton sector to test the BSMs for the neutrino mixing.