Pirfenidone and nintedanib are the only two drugs approved for the treatment of idiopathic pulmonary fibrosis (IPF). Both proved to be safe and well-tolerated in clinical trials, but real-world data and direct comparisons are scarce. This real-life study explored the safety profile of pirfenidone and nintedanib with a prolonged follow-up. We retrospectively collected clinical status, adverse events (AEs), and treatment changes from IPF patients who had started an antifibrotic treatment at our centre from December 2011 to December 2020, including 192 patients treated with pirfenidone and 89 with nintedanib. The majority of patients in both groups experienced one or more AEs during the follow-up. A higher proportion of AEs in the nintedanib group were effectively treated with behavioural modifications or additional medications compared with the pirfenidone group (52.5% vs. 40.6%, p = 0.04). Overall, a difference in the impact of AEs due to nintedanib versus pirfenidone resulted in a lower permanent discontinuation of therapy (8.3% vs. 18.3%, p = 0.02), with the latter being associated with a higher risk of drug discontinuation at 48 months after initiation (OR =2.52, p = 0.03). Our study confirms the safety profile of antifibrotic drugs in IPF but highlights that AEs due to nintedanib are usually easier to manage and lead to fewer cases of permanent discontinuation of therapy.