The Omicron subvariant BA.2 caused the start of the fifth epidemic wave in Hong Kong in early 2022, leading to a significant outbreak and triggering more people get immunized in a population with relatively low vaccine coverage. About half a year later, a second outbreak, largely dominated by BA.4 and BA.5 subvariants, began to spread, which peaked within few months. How the waning of the vaccine protection and the immune escape properties together with other factors, including social distancing and outdoor temperature, drove the second surge of infections is unknown. The challenge is that basic epidemic modeling is not able to capture longitudinal change in vaccine waning and dose interval. We developed mathematical equations to formulate continuous change of vaccine waning after observing empirical serological data and incorporate them into a multi-strain discrete-time SEIR (Susceptible-Exposed-Infectious-Removed) model. Using the reported cases during the first outbreak as the training set together with daily vaccination rates, population mobility and temperature, the model successfully predicted the second surge and the replacement by BA.4/5, leading to a cumulative number of cases about 543,600 (7.27% of total population). If vaccine protection maintained without decreasing, the number of predicted cumulative cases reduced 35%. If perfect vaccine coverage reached (100%) by 1st June, the number of cases was only partially reduced 18.65%. Moderate level of social distancing (10% reduction in population mobility) reduced only 13.78% cases, which was not able to prevent the second surge. The results suggest future local outbreaks will remain as long as new immune escape happens with waning immunity, even if the moderate level of social distancing is present. Therefore, a more accurate model forecasting considering waning immunity remains needed in order to allow a better preparation of the next outbreaks.