We study the modular invariance in magnetized torus models. The modular invariant flavor model is a recently proposed hypothesis for solving the flavor puzzle, where the flavor symmetry originates from modular invariance. In this framework, coupling constants such as Yukawa couplings are also transformed under the flavor symmetry. We show that the low-energy effective theory of magnetized torus models is invariant under a specific subgroup of the modular group. Since Yukawa couplings as well as chiral zero modes transform under the modular group, the above modular subgroup (referred to as modular flavor symmetry) provides a new type of modular invariant flavor models with D 4 × Z 2 , ðZ 4 × Z 2 Þ ⋊ Z 2 , and ðZ 8 × Z 2 Þ ⋊ Z 2. We also find that conventional discrete flavor symmetries which arise in magnetized torus model are noncommutative with the modular flavor symmetry. Combining both symmetries, we obtain a larger flavor symmetry, which is the semidirect product of the conventional flavor symmetry and the modular flavor symmetry for the nonvanishing Wilson line. For the vanishing Wilson line, we have additional Z 2 symmetry, i.e., parity, which is the unique common element between the conventional flavor symmetry and the modular flavor symmetry.