ONEST (Observers Needed to Evaluate Subjective Tests) suggests four or more observers for a reliable assessment of the consistency of histological grading of invasive breast carcinoma: A reproducibility study with a retrospective view on previous studies

ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations

“…ONEST is a recently described additional analysis that can complement reproducibility studies [11][12][13]. Although it was introduced to estimate the minimum number of observers required to provide a reliable estimate of the reproducibility of a given classification [11], it also gives a visual impression of how much agreement is reached when categorising items into predefined classes and the difference one can expect between two observers.…”

“…There were 2 missing values in all circulations (C1, C1s, and C2) which were replaced by mean sTIL percentages rounded to the closest integer. For the ONEST analysis, as per the introduction of the method and its subsequent uses [11][12][13], 100 randomly selected permutations were selected for the values of the ONEST plots. Four selected cut-offs were used to define categories: <60% vs. ≥60%, e.g., [14], and <50% vs. ≥50%, e.g., [15,16], to match two different definitions of lymphocyte-predominant breast cancers, which are the likeliest responders to neoadjuvant treatment [6]; <30% vs. ≥30% to match a cut-off proposed for a strong prognostic role in the adjuvant setting [3]; and 0-20%, 21-49%, and ≥50% to match a three-tiered classification used in the IIOBWG ring studies [8].…”

“…In a previous study, 9 pathologists assessed the ER, the progesterone receptor (PR) status, Ki67 labelling [12], and histological grade [13] of breast cancers in 50 core needle biopsies and 50 resection specimens represented on a full-face glass slide for each case. While assessing these parameters, the participants were also asked to document sTILs based on the IIOBMWG recommendations [6,7], which are also part of the Hungarian recommendation [17,18].…”

“…ONEST is a recently developed method that complements inter-observer agreement studies by helping to estimate the number of observers required for a reliable estimation of reproducibility [11]. ONEST uses 100 randomly selected permutations of all participating pathologists (observers or raters) and plots the overall percent agreement (OPA) values for an increasing number of observers, looking for the worst (lowest) curve to reach a plateau, beyond which an increasing number of observers does not have a substantial effect on agreement [11][12][13]. Additionally, ONEST has been recognised to be valuable as a visual complement to demonstrate the degree of reproducibility of subjectively evaluated parameters such as oestrogen receptor (ER) quantification, Ki-67 labelling, or histological grade, as well as the difference between observers and how these compare to the overall percent agreement (OPA) of all observers [12,13].…”

“…ONEST uses 100 randomly selected permutations of all participating pathologists (observers or raters) and plots the overall percent agreement (OPA) values for an increasing number of observers, looking for the worst (lowest) curve to reach a plateau, beyond which an increasing number of observers does not have a substantial effect on agreement [11][12][13]. Additionally, ONEST has been recognised to be valuable as a visual complement to demonstrate the degree of reproducibility of subjectively evaluated parameters such as oestrogen receptor (ER) quantification, Ki-67 labelling, or histological grade, as well as the difference between observers and how these compare to the overall percent agreement (OPA) of all observers [12,13]. The aim of this study is to evaluate sTIL quantification using ONEST and to estimate the number of observers needed for a reliable evaluation of its reproducibility.…”