We present a study of the thermodynamic and magnetic properties of single-crystal EuTiO3. Signatures of metastability are visible in the heat capacity below the cubic-tetragonal phase transition at 283 K, supporting the evidence for a mismatch between long and short range structural order from previous X-ray diffraction studies. Employing the anisotropic magnetization as an indirect structural probe, we confirm the emergence of multiple orthogonal domains at low temperature. Torque magnetometry is capable of revealing the nature and temperature-dependence of the magnetic anisotropy in spite of the domain misalignment; we hence deduce that tetragonal EuTiO3 enters an easy-axis antiferromagnetic phase at 5.6 K, with a first-order phase transition to an easyplane ground state below 3 K. Our experimentally-determined magnetic phase diagram is accurately reproduced by a 3D anisotropic Heisenberg spin model. Furthermore, we demonstrate that electric field cooling acts to suppress this orientational disorder by realigning the domains, due to the strong coupling between electric fields and lattice dipoles characteristic of paraelectric materials.