We have investigated the modular binary octahedral group 2O as a flavor symmetry to explain the structure of Standard Model. The vector-valued modular forms in all irreducible representations of this group are constructed. We have classified all possible fermion mass models based on the modular binary octahedral group 2O. A comprehensive numerical analysis is performed, and we present some benchmark quark/lepton mass models in good agreement with the experimental data. Notably we find a minimal modular invariant model for leptons and quarks, which is able to explain simultaneously the masses and mixing parameters of both quarks and leptons in terms of 14 real free parameters including the modulus τ. The fermion mass hierarchies around the vicinity of the modular fixed points are explored.