Phenomenological studies of Flavored Dark Matter (FDM) models often have to assume a near-diagonal flavor structure in the coupling matrix in order to remain consistent with bounds from flavor violating processes. In this paper we show that for Lepton FDM, such a structure can naturally arise from an extra dimensional setup. The extra dimension is taken to be flat, with the dark matter and mediator fields confined to a brane on one end of the extra dimension, and the Higgs field to a brane on the other end. The Standard Model fermion and gauge fields are the zero modes of corresponding bulk fields with appropriate boundary conditions. Global flavor symmetries exist in the bulk and on the FDM brane, while they are broken on the Higgs brane. Flavor violating processes arise due to the misalignment of bases for which the interactions on the two branes are diagonalized, and their size can be controlled by a choice of the lepton profiles along the extra dimension. By studying the parameter space for the model, we show that when relic abundance and indirect detection constraints are satisfied, the rates for flavor violating processes such as µ → eγ remain far below the experimental limits.