Universal Metrics for Quality Assessment of Protein Identifications by Mass Spectrometry

Abstract

Increasing numbers of large proteomic datasets are becoming available. As attempts are made to interpret these datasets and integrate them with other forms of genomic data, researchers are becoming more aware of the importance of data quality with respect to protein identification. We present three simple and universal metrics that describe different aspects of the quality of protein identifications by peptide mass fingerprinting. Hit ratio gives an indication of the signal-to-noise ratio in a mass spectrum, mass coverage measures the amount of protein sequence matched, and excess of limit-digested peptides reflects the completeness of the digestion that precedes the peptide mass fingerprinting. Receiver-operating characteristic plots show that the novel metric, excess of limit-digested peptides, can discriminate between correct and random matches more accurately than search score when validating the results from a state-of-the-art protein identification software system (Mascot) especially when combined with the two other metrics, hit ratio and mass coverage. Recommendations are made regarding the use of the metrics when reporting protein identification experiments.

Iterative Non‐m/z‐sharing Rule for Confident and Sensitive Protein Identification of Non‐shotgun Proteomics

“…The researchers represented the results with receiver operating characteristic (ROC) plot, which displayed the ratio of true positive classifications to the fraction of false positives (1-specificity) as a function of discriminatory performance. 33,34 The larger area under the curve indicates the better performance of discrimination. Herein, the confirmed results of each strategy were defined as the correct matches, whereas the highest ranked proteins in CRRSDB were taken as the incorrect matches.…”

“…33 It has been reported that ELDP can discriminate between correct and random matches more accurately than the search score of MASCOT. The researchers represented the results with receiver operating characteristic (ROC) plot, which displayed the ratio of true positive classifications to the fraction of false positives (1-specificity) as a function of discriminatory performance.…”

Managing information quality in e-science

“…Based on the ranking, data acceptability decisions can be made by choosing suitable thresholds on the ordered set. Such a score model for protein hits has indeed been proposed by a group of collaborators at the Aberdeen Proteomics Facility [6]. The model has two interesting properties, namely (i) it shows excellent true positive-to-false positive ratio performance on extensive test sets; and (ii) it is computed using parameters that are common to most available PMF matching algorithms, regardless of their proprietary nature.…”

Exploiting Provenance to Make Sense of Automated Decisions in Scientific Workflows

“…Recent research [11] has shown a strong correlation between a simple score model for matches based on this and other readily available indicators, and the likelihood of false positives. By using this predictive score model to rank the output of the matching service, in combination with a user-defined threshold, experimenters have an effective way to make their quality acceptance criteria formal and automatically computable.…”