Many-body dipolar effects in Fermi gases are quite subtle as they energetically compete with the large kinetic energy at and below the Fermi surface (FS). Recently it was experimentally observed in a sample of erbium atoms that its FS is deformed from a sphere to an ellipsoid due to the presence of the anisotropic and long-range dipole-dipole interaction Aikawa et al (2014 Science 345 1484). Moreover, it was suggested that, when the dipoles are rotated by means of an external field, the FS follows their rotation, thereby keeping the major axis of the momentum-space ellipsoid parallel to the dipoles. Here we generalise a previous Hartree-Fock mean-field theory to systems confined in an elongated triaxial trap with an arbitrary orientation of the dipoles relative to the trap. With this we study for the first time the effects of the dipoles' arbitrary orientation on the ground-state properties of the system. Furthermore, taking into account the geometry of the system, we show how the ellipsoidal FS deformation can be reconstructed, assuming ballistic expansion, from the experimentally measurable real-space aspect ratio after a free expansion. We compare our theoretical results with new experimental data measured with erbium Fermi gas for various trap parameters and dipole orientations. The observed remarkable agreement demonstrates the ability of our model to capture the full angular dependence of the FS deformation. Moreover, for systems with even higher dipole moment, our theory predicts an additional unexpected effect: the FS does not simply follow rigidly the orientation of the dipoles, but softens showing a change in the aspect ratio depending on the dipoles' orientation relative to the trap geometry, as well as on the trap anisotropy itself. Our theory provides the basis for understanding and interpreting phenomena in which the investigated physics depends on the underlying structure of the FS, such as fermionic pairing and superfluidity.