Motivation:Publicly available mass spectrometry-based proteomics data has grown exponentially in the recent past. Yet, large scale spectrum-centered analysis usually involves predefined fragmentation features that are limited and prone to be biased. Using deep learning, the decision making for a suitable fragmentation model can be carried out in a data-driven manner. Results: We introduce a framework that allows end-to-end training of generic deep learning models on a large collection of high resolution tandem mass spectra. In this case we used 19.2 million labeled spectra from more than a hundred individual PRIDE repositories. In our framework, we developed a representation that captures the complete information of a high-resolution spectrum facilitating a loss-less reduction of the number of features largely independent of the actual resolution. Additionally, it allows us to use common trainable layers, e.g. recurrent or convolutional operations. Specifically, we use a deep network of stacked dilated convolutions to model long range associations between any peaks within a tandem mass spectrum. We exemplify our approach by learning to detect post-translational modifications -in this case, protein phosphorylation -only based on a given mass spectrum in a fully data-driven manner. To the best of our knowledge, this is the first end-to-end trained deep learning model on tandem spectra that is able to ad hoc learn fragmentation patterns in high-resolution spectra. Our approach outperforms the current state-of-the-art in predicting if a mass spectrum originates from a phosphorylated peptide. Availability: Our deep learning framework is implemented in tensorflow. The open source code including trained weights is available at gitlab.com/dacs-hpi/ahlf Contact: bernhard.renard@hpi.de