We have studied the lepton and quark mixing patterns which can be derived from the dihedral group D n in combination with CP symmetry. The left-handed lepton and quark doublets are assigned to the direct sum of a singlet and a doublet of D n. A unified description of the observed structure of the quark and lepton mixing can be achieved if the flavor group D n and CP are broken to Z 2 × CP in neutrino, charged lepton, up quark and down quark sectors, and the minimal group is D 14. We also consider another scenario in which the residual symmetry of the charged lepton and up quark sector is Z 2 while Z 2 ×CP remains preserved by the neutrino and down quark mass matrices. Then D 7 can give the experimentally favored values of CKM and PMNS mixing matrices.