Epicardial adipose tissue (EAT) and pericardial adipose tissue (PAT) contribute to the development of left atrial fibrillation (AF). The purpose of this study is to determine the factors influencing cardiac fat, evaluate its impact on heart function, and evaluate its role in the recurrence of AF. Cardiac MRI exams of n = 198 patients with paroxysmal AF were retrospectively analyzed to quantify EAT and PAT. Body mass index (BMI) showed significant associations with increased EAT, PAT, and total cardiac fat, particularly with the total end-systolic area (p < 0.001). Males were associated with increased PAT (r = −0.331, p < 0.001) and EAT (r = −0.168, p = 0.019). Increased PAT end-diastolic volume was also associated with an increase in LV mass (r = 0.249, p < 0.01). An inverse relationship between EAT end-systolic area and cardiac index (r = −0.220, p < 0.01) was observed. Although BMI did not significantly affect AF recurrence, overweight patients (36%) experienced slightly more AF recurrence than obese patients (33%). Obesity is substantially associated with an increase in EAT and PAT, while sex appears to play a greater role in PAT than EAT and decreased cardiac function.