The current understanding of activity in the wireless spectrum is limited to mostly punctual studies of aggregated energy values. However, there is a need and increasing technological means for a better understanding of spectrum usage by automatically detecting and recognizing wireless transmissions in an unlicensed or shared frequency band. In this paper we propose, implement and evaluate a framework for automatic detection of wireless transmissions. Our framework includes a manual component as our assessment suggests manual labor has a paramount impact on tuning and maintaining good performance of an automatic transmission detection system. However, a considerable problem in this aspect is represented by the disagreement amongst human annotations which is a universally recognized issue. To this end, we discuss and evaluate challenges in generating labeled datasets that can then be used as ground truth for evaluating and possibly training automatic transmission detection systems. We also propose two methods for automatic transmission detection that are not based on machine learning and therefore do not need training data and evaluate their performance against each other and manually labeled data. Our results show that generating human-labeled ground truth data is an expensive and imperfect process. Humans on average require 90 minutes to label 56 minutes of unlicensed European narrowband spectrum. The experts that generate the ground truth sometimes only agree on as little as 40.18% of the labeled cases.