Antibiotic-induced dysbiosis may affect the efficacy of immune checkpoint inhibitors. We investigated the impact of antibiotics on the clinical outcomes of nivolumab in patients with non-small cell lung cancer (NSCLC). Patients who received nivolumab for NSCLC between July 2015 and June 2018 and who were followed up until June 2020 were included in a retrospective cohort analysis. Of 140 eligible patients, 70 were on antibiotics. Overall survival (OS) was shorter in patients on antibiotics (ABX) compared to those not on antibiotics (NoABX) (p = 0.014). OS was negatively associated with piperacillin/tazobactam (PTZ) (HR = 3.31, 95% CI: 1.77–6.18), days of therapy (DOT) ≥ 2 weeks (HR = 2.56, 95% CI: 1.30–5.22) and DOT of PTZ. The defined daily dose (DDD) in PTZ (r = 0.27) and glycopeptides (r = 0.21) showed weak correlations with mortality. There was no difference in progression-free survival (PFS) between ABX and NoABX; however, PFS was negatively associated with the antibiotic class PTZ and DOT of PTZ. Therefore, the use of a broad-spectrum antibiotic, such as PTZ, the long-term use of antibiotics more than 2 weeks in total and the large amount of defined daily dose of specific antibiotics were associated with decreased survival in patients receiving nivolumab for NSCLC.