Superconducting qubit designs vary in complexity from single-and few-junction systems, such as the transmon and flux qubits, to the many-junction fluxonium. Here, we consider the question of whether the many degrees of freedom in the fluxonium circuit can limit the qubit coherence time. Such a limitation is in principle possible, due to the interactions between the low-energy, highly anharmonic qubit mode and the higher-energy, weakly anharmonic collective modes. We show that so long as the coupling of the collective modes with the external electromagnetic environment is sufficiently weaker than the qubit-environment coupling, the qubit dephasing induced by the collective modes does not significantly contribute to decoherence. Therefore, the increased complexity of the fluxonium qubit does not constitute by itself a major obstacle for its use in quantum computation architectures.