Spectral alignment of tandem mass spectrometry (MS/MS) data using the modified cosine similarity and subsequent visualization as molecular networks have been demonstrated to be a useful strategy to discover analogs of molecules from untargeted MS/MS-based metabolomics experiments. Recently, a neutral loss matching approach has been introduced as an alternative to MS/MS-based molecular networking, with an implied performance advantage in finding analogs that cannot be discovered using existing MS/MS spectral alignment strategies. To comprehensively evaluate the scoring properties of neutral loss matching, the cosine similarity, and the modified cosine similarity, similarity measures of 955,228 peptide MS/MS spectrum pairs and 10 million small molecule MS/MS spectrum pairs were compared. This comparative analysis revealed that the modified cosine similarity outperformed neutral loss matching and the cosine similarity in all cases. The data further indicated that the performance of neutral loss matching depends on both the location and type of the modification. For full transparency and reproducibility, and reusability for any future comparisons of newly developed mass spectral alignment approaches, all code and benchmark data that was used in this work are freely accessible to the community.