Deep learning models are now widely used in decision-making applications. These models must be robust to noise and carefully map to the underlying uncertainty in the data. Standard deterministic neural networks are well known to be poor at providing reliable estimates of uncertainty and often lack the robustness that is required for real-world deployment. In this paper, we work with an application that requires accurate uncertainty estimates in addition to good predictive performance. In particular, we consider the task of detecting a mosquito from its acoustic signature. We use Bayesian neural networks (BNNs) to infer predictive distributions over outputs and incorporate this uncertainty as part of an automatic labelling process. We demonstrate the utility of BNNs by performing the first fully automated data collection procedure to identify acoustic mosquito data on over 1,500 hours of unlabelled field data collected with low-cost smartphones in Tanzania. We use uncertainty metrics such as predictive entropy and mutual information to help with the labelling process. We show how to bridge the gap between theory and practice by describing our pipeline from data preprocessing to model output visualisation. Additionally, we supply all of our data and code. The successful autonomous detection of mosquitoes allows us to perform analysis which is critical to the project goals of tackling mosquito-borne diseases such as malaria and dengue fever.