Background and objectives: Pulmonary vein (PV) reconnection is a major reason for recurrence after catheter ablation of paroxysmal atrial fibrillation (PAF). However, the timing of the recurrence varies between patients, and recurrence >1 year after ablation is not uncommon. We sought to elucidate the characteristics of atrial fibrillation (AF) that recurred in different follow-up periods. Materials and Methods: Study subjects comprised 151 consecutive patients undergoing initial catheter ablation of PAF. Left atrial volume index (LAVi) and atrial/brain natriuretic peptide (ANP/BNP) levels were systematically measured annually over 3 years until AF recurred. Results: Study subjects were classified into four groups: non-recurrence group (n = 84), and short-term- (within 1 year) (n = 30), mid-term- (1–3 years) (n = 26), and long-term-recurrence group (>3 years) (n = 11). The short-term-recurrence group was characterized by a higher prevalence of diabetes mellitus (hazard ratio 2.639 (95% confidence interval, 1.174–5.932), p = 0.019 by the Cox method), frequent AF episodes (≥1/week) before ablation (4.038 (1.545–10.557), p = 0.004), and higher BNP level at baseline (per 10 pg/mL) (1.054 (1.029–1.081), p < 0.0001). The mid-term-recurrence group was associated with higher BNP level (1.163 (1.070–1.265), p = 0.0004), larger LAVi (mL/m2) (1.033 (1.007–1.060), p = 0.013), and longer AF cycle length at baseline (per 10 ms) (1.194 (1.058–1.348), p = 0.004). In the long-term-recurrence group, the ANP and BNP levels were low throughout follow-up, as with those in the non-recurrence group, and AF cycle length was shorter (0.694 (0.522–0.924), p = 0.012) than those in the other recurrence groups. Conclusions: Distinct characteristics of AF were found according to the time to first recurrence after PAF ablation. The presence of secondary factors beyond PV reconnections could be considered as mechanisms for the recurrence of PAF in each follow-up period.