Objective: We aimed to study the long-term association of LV mass index (LVMI) and myocardial fibrosis with ventricular arrhythmia (VA) in a population of patients with confirmed hypertrophic cardiomyopathy (HCM) using cardiac magnetic resonance imaging (CMR). Methods: We retrospectively analyzed the data in consecutive HCM patients confirmed on CMR referred to an HCM clinic between January 2008 and October 2018. Patients were followed up yearly following diagnosis. Baseline demographics, risk factors and clinical outcomes from cardiac monitoring and an implanted cardioverter defibrillator (ICD) were analyzed for association of LVMI and LV late gadolinium enhancement (LVLGE) with VA. Patients were then allocated to one of two groups according to the presence of VA (Group A) or absence of VA (Group B) during the follow-up period. The transthoracic echocardiogram (TTE) and CMR parameters were compared between the two groups. Results: A total of 247 patients with confirmed HCM (age 56.2 ± 16.6, male = 71%) were studied over the follow-up period of 7 ± 3.3 years (95% CI = 6.6–7.4 years). LVMI derived from CMR was higher in Group A (91.1 ± 28.1 g/m2 vs. 78.8 ± 28.3 g/m2, p = 0.003) when compared to Group B. LVLGE was higher in Group A (7.3 ± 6.3% vs. 4.7 ± 4.3%, p = 0.001) when compared to Group B. Multivariable Cox regression analysis showed LVMI (hazard ratio (HR) = 1.02, 95% CI = 1.001–1.03, p = 0.03) and LVLGE (HR = 1.04, 95% CI = 1.001–1.08, p = 0.04) to be independent predictors for VA. Receiver operative curves showed higher LVMI and LVLGE with a cut-off of 85 g/m2 and 6%, respectively, to be associated with VA. Conclusions: LVMI and LVLGE are strongly associated with VA over long-term follow-up. LVMI requires more thorough studies to consider it as a risk stratification tool in patients with HCM.