There is a large gap between primitive bruxism detectors and sophisticated clinical machines for jaw kinematics evaluation. Large, expensive clinical appliances can precisely record jaw motion, but completely restrain the patient for the duration of the test. Wearable bruxism detectors allow continuously counting and recording bites, but provide no information about jaw movement trajectories. Previously, we developed a permanent magnet and three-axis magnetometer-based method for wearable, intra-oral continuous jaw position registration. In this work, we present an effective solution of the two main drawbacks of the method. Firstly, a two-adjacent-magnetometer approach is able to compensate for background magnetic fields with no reference sensor outside of the system’s magnetic field. Secondly, jaw rotational angles were included in the position calculations, by applying trigonometric equations that link the translation of the jaw to its rotation. This way, we were able to use a three-degree-of-freedom (3-DOF) magnetic position determination method to track the positions of the 5-DOF human masticatory system. To validate the method, finite element modeling and a 6-DOF robotic arm (0.01 mm, 0.01°) were used, which showed a 37% decrease in error in the average RMSE = 0.17 mm. The method’s potentially can be utilized in small-scale, low-power, wearable intra-oral devices for continuous jaw motion recording.