While the first infection by an emerging disease is often unknown, information on early cases can be used to date it, which is of great interest to trace the disease's origin and understand early infection dynamics. In the context of the COVID-19 pandemic, previous studies have estimated the date of emergence (e.g., first human SARS-CoV-2 infection in Wuhan, emergence of the Alpha variant in the UK) using mainly genomic data. Another dating attempt only relied on case data, estimating a date of emergence using a non-Markovian stochastic model and considering the first case detection. Here, we extend this stochastic approach to use available data of the whole early case dynamics. Our model provides estimates of the delay from the first infection to the Nthreported case. We first validate our model using data concerning the spread of the Alpha SARS-CoV-2 variant in the UK. Our results suggest that the first Alpha infection occurred on (median) August 20 (95% interquantile range across retained simulations, IqR: July 20-September 4), 2020. Next, we apply our model to data on the early reported cases of COVID-19. We used data on the date of symptom onset up to mid-January, 2020. We date the first SARS-CoV-2 infection in Wuhan at (median) November 26 (95%IqR: October 31-December 7), 2019. Our results fall within ranges previously estimated by studies relying on genomic data. Our population dynamics-based modelling framework is generic and flexible, and thus can be applied to estimate the starting time of outbreaks, in contexts other than COVID-19, as long as some key parameters (such as transmission and detection rates) are known.