Temporomandibular joint osteoarthritis (TMJ OA) is a disease with a multifactorial etiology, involving many pathophysiological processes, and requiring comprehensive assessments to characterize progressive cartilage degradation, subchondral bone remodeling, and chronic pain. This study aimed to integrate quantitative biomarkers of bone texture and morphometry of the articular fossa and joint space to advance the role of imaging phenotypes for diagnosis of Temporomandibular Joint Osteoarthritis (TMJ OA) in early to moderate stages by improving the performance of machine-learning algorithms to detect TMJ OA status. Ninety-two patients were prospectively enrolled (184 h-CBCT scans of the right and left mandibular condyles), divided into two groups: 46 control and 46 TMJ OA subjects. No significant difference in the articular fossa radiomic biomarkers was found between TMJ OA and control patients. The superior condyle-to-fossa distance (p < 0.05) was significantly smaller in diseased patients. The interaction effects of the articular fossa radiomic biomarkers enhanced the performance of machine-learning algorithms to detect TMJ OA status. The LightGBM model achieved an AUC 0.842 to diagnose the TMJ OA status with Headaches and Range of Mouth Opening Without Pain ranked as top features, and top interactions of VE-cadherin in Serum and Angiogenin in Saliva, TGF-β1 in Saliva and Headaches, Gender and Muscle Soreness, PA1 in Saliva and Range of Mouth Opening Without Pain, Lateral Condyle Grey Level Non-Uniformity and Lateral Fossa Short Run Emphasis, TGF-β1 in Serum and Lateral Fossa Trabeculae number, MMP3 in Serum and VEGF in Serum, Headaches and Lateral Fossa Trabecular spacing, Headaches and PA1 in Saliva, and Headaches and BDNF in Saliva. Our preliminary results indicate that condyle imaging features may be more important in regards to main effects, but the fossa imaging features may have a larger contribution in terms of interaction effects. More studies are needed to optimize and further enhance machine-learning algorithms to detect early markers of disease, improve prediction of disease progression and severity to ultimately better serve clinical decision support systems in the treatment of patients with TMJ OA.