Background and Objectives: Since the start of the COVID-19 pandemic caused by severe acute respiratory syndrome coronavirus II, levels of physical inactivity have become more severe and widespread than ever before. Physical inactivity is known to have a negative effect on the human body, but the extent to which reduced physical fitness has effected immune function before and after the current pandemic has not yet been uncovered. The aim of this study was to investigate the detraining effects of the COVID-19 confinement period on physical fitness, immunocytes, inflammatory cytokines, and proteins in various age groups. The participants of this study included sixty-four male adults who did not exercise during the pandemic, although they had exercised regularly before. Materials and Methods: Participants were classified by age group, which included the 20s group (20s’G, n = 14), 30s group (30s’G, n = 12), 40s group (40s’G, n = 12), 50s group (50s’G, n = 12), and 60s group (60s’G, n = 14). Results: Regarding body composition, muscle mass significantly decreased, whereas fat mass, fat percentage, and waist/hip ratio significantly increased in most groups. Cardiopulmonary endurance and strength significantly decreased in all groups, while muscle endurance and flexibility decreased in some groups compared to the pre-COVID-19 pandemic. This study confirmed the immunocytopenia and enhanced inflammation due to physical inactivity during the COVID-19 pandemic, and a greater detrimental decrease mainly after the age of 50. Conclusion: This study confirmed a decrease in physical fitness after the start of the COVID-19 pandemic, characterized by an increase in fat mass and a decrease in muscle mass, thereby increasing cytokines and reducing immunocytes in the body. While social distancing is important during the pandemic, maintaining physical fitness should also be a top priority.