We explore alternative descriptions of the charged lepton sector in modular invariant models of lepton masses and mixing angles. In addition to the modulus, the symmetry breaking sector of our models includes ordinary flavons. Neutrino mass terms depend only on the modulus and are tailored to minimize the number of free parameters. The charged lepton Yukawa couplings rely upon the flavons alone. We build modular invariant models at levels 4 and 5, where neutrino masses are described both in terms of the Weinberg operator or through a type I seesaw mechanism. At level 4, our models reproduce the hierarchy among electron, muon and tau masses by letting the weights play the role of Froggatt-Nielsen charges. At level 5, our setup allows the treatment of left and right handed charged leptons on the same footing. We have optimized the free parameters of our models in order to match the experimental data, obtaining a good degree of compatibility and predictions for the absolute neutrino masses and the CP violating phases. At a more fundamental level, the whole lepton sector could be correctly described by the simultaneous presence of several moduli. Our examples are meant to make a first step in this direction. arXiv:1908.11867v1 [hep-ph] 30 Aug 2019 Recently, modular invariance has been invoked as candidate flavour symmetry [10]. In its simplest implementation a unique complex field, the modulus, acts as symmetry breaking parameter, thus simplifying the vacuum alignment problem. Modular invariance, in the limit of exact supersymmetry, completely determines the Yukawa couplings, to any order of the expansion in powers of the modulus. Moreover, neutrino masses, mixing angles and phases are all related to each other and, in minimal models, depend only on a few parameters. The formalism has been extended to consistently include CP transformations [11] 1 and it can involve several moduli [16,17]. The idea that Yukawa couplings are determined by a set of moduli is clearly not new, and has been naturally realized in the context of string theory [18][19][20][21][22], in D-brane compactification [23][24][25][26][27][28][29], in magnetized extra dimensions [30][31][32], and in orbifold compactification [33][34][35][36]. Modular invariance has also been incorporated in early flavour models [37][38][39][40][41]. However, the main advantage of the recent approach is that it can be implemented in a bottom-up perspective, relying on the group transformation properties of modular forms of given weight and level. Several models of lepton masses and mixing angles have been built at level 2 [42, 43], 3 [10, 44-47], 4 [48-50] and 5 [51,52]. Extensions to quarks [53,54] and to grand unified theories [55,56] have also been proposed. In most of the existing constructions, there is a unique symmetry breaking parameter: the modulus itself. While this scenario is certainly appealing since it minimizes the symmetry breaking sector, it does not yet provide a convincing explanation of the charged lepton masses. The mass hierarchy is achieved by...