Space group symmetries dictate the energy degeneracy of quasiparticles (e.g., electronic, photonic) in crystalline structures. For spinless systems, there can only be double or triple degeneracies protected by these symmetries, while other degeneracies are usually taken as accidental. In this Letter we show that it is possible to design higher degeneracies exploring site permutation symmetries. These design principles are shown to be satisfied in previously studied lattices, and new structures are proposed with three, four and five degeneracy points for spinless systems. The results are general and apply to different quasiparticle models. Here, we focus on a tight-binding approach for the electronic case as a proof of principle. The resulting high-degeneracy points are protected by the site-permutation symmetries, yielding pseudospin-1 and -2 Dirac fermions. The strategy proposed here can be used to design lattices with high-degeneracy points in electronic (e.g. metal-organic frameworks), photonic, phononic, magnonic and cold-atom systems.