We assessed the antibiotic use in SARS-CoV-2-infected patients during four different waves of the COVID-19 pandemic, as well as its trends over the period and associated risk factors. We performed a cross-sectional retrospective analysis nested in a prospectively collected cohort of hospitalized adult patients with COVID-19 at a university hospital in Spain. A total of 2415 patients were included in this study, among whom 1120 corresponded to the first wave. The highest percentage of patients receiving some sort of antibiotic treatment was higher during the first wave (77.6%) than during the others; nevertheless, our calculation of the average DOT (days of antibiotic treatment) per 100 patient days of stay found that the highest antibiotic prescription rate corresponded to the second pandemic wave (61.61 DOT/100 patient days), which was associated with a higher ICU admission rate and a lower SpO2/FiO2 ratio at admission. After the second wave, the prescription rates presented a steady downward trend. With regard to the use of specific antibiotic families, amoxicillin/clavulanate was the most used antibiotic in our cohort (14.20 DOT/100 patient days) due to a high prescription rate during the first wave. According to the “AWaRe” WHO classification, antibiotics corresponding to the “Watch” group were the most prescribed (27.92 DOT/100 patient days). The antibiotic use rate fell progressively, but it remained high during all four waves analyzed. In conclusion, antibiotic use was high throughout all the waves that were analyzed, despite a relatively low incidence of bacterial coinfection and superinfection. Efforts should be made to keep antimicrobial stewardship programs active, especially in complicated epidemiological situations, such as the SARS-CoV-2 pandemic.