Objectives: Observational studies have shown the association between knee osteoarthritis (KOA) and neurological disorders with alterations in brain imaging-derived phenotypes (BIDPs). This study aimed at investigating whether alterations in brain structure are correlated with the occurrence of KOA. Methods: Based on the summary data from two large scale genome-wide association studies (GWASs), we performed a bidirectional two-sample Mendelian randomization (MR) analysis using single-nucleotide polymorphisms (SNPs) as instrumental variables (IVs) to determine the potential causal relationships between KOA and BIDPs. Results: We identified the genetic correlations of 152 BIDPs with KOA using linkage disequilibrium score regression. MR analysis revealed that increased volume but decreased intensity-contrast of bilateral nucleus accumbens (NAc), as well as increased left paracentral area was positively causally associated with KOA risk. For the IDPs of structural connectivity, we identified causal associations between multiple increased DTI parameter indicators of corticospinal tract (CST) and KOA. Inversely, KOA was positively correlated with the thickness and intensity-contrast of the rostral anterior cingulate, as well as the intensity-contrast of caudal anterior cingulate, insula cortex, and the grey matter volume of pallidum. Conclusion: Our study supported bidirectional causal associations between KOA and BIDPs, which may provide new insights into the interaction of KOA with structural alterations in the nervous system.