Reproducibility of the Development and Validation Process of Standard Area Diagram by Two Laboratories: An Example Using the Botrytis cinerea/Gerbera jamesonii Pathosystem

Abstract

<jats:p> Standard area diagrams (SADs) are plant disease severity assessment aids demonstrated to improve the accuracy and reliability of visual estimates of severity. Knowledge of the sources of variation, including those specific to a lab such as raters, specific procedures followed including instruction, image analysis software, image viewing time, etc., that affect the outcome of development and validation of SADs can help improve standard operating practice of these assessment aids. As reproducibility has not previously been explored in development of SADs, we aimed to explore the overarching question of whether the lab in which the measurement and validation of a SAD was performed affected the outcome of the process. Two different labs (Lab 1 and Lab 2) measured severity on the individual diagrams in a SAD and validated them independently for severity of gray mold (caused by Botrytis cinerea) on Gerbera daisy. Severity measurements of the 30 test images were performed independently at the two labs as well. A different group of 18 raters at each lab assessed the test images first without, and secondly with SADs under independent instruction at both Lab 1 and 2. Results showed that actual severity on the SADs as measured at each lab varied by up to 5.18%. Furthermore, measurement of the test image actual values varied from 0 to up to 24.29%, depending on image. Whereas at Lab 1 an equivalence test indicated no significant improvement in any measure of agreement with use of the SADs, at Lab 2, scale shift, generalized bias, and agreement were significantly improved with use of the SADs (P ≤ 0.05). An analysis of variance indicated differences existed between labs, use of the SADs aid, and the interaction, depending on the agreement statistic. Based on an equivalence test, the interrater reliability was significantly (P ≤ 0.05) improved at both Lab 1 and Lab 2 as a result of using SADs as an aid to severity estimation. Gain in measures of agreement and reliability tended to be greatest for the least able raters at both Lab 1 and Lab 2. Absolute error was reduced at both labs when raters used SADs. The results confirm that SADs are a useful tool, but the results demonstrated that aspects of the development and validation process in different labs may affect the outcome. </jats:p>

Severity assessment in the Nicotiana tabacum-Xylella fastidiosa subsp. pauca pathosystem: design and interlaboratory validation of a standard area diagram set

“…Although a plethora of software has been used to determine actual severity (Del Ponte et al 2017), comparison among the different software programs is not often an objective in SADs studies. Variability in severity measurements on the same specimen with different software programs or raters has been reported (Bock et al 2008;Melo et al 2020). Therefore, any of the software programs tested here can be used if the conditions are the same, especially if one trained operator is thresholding the images.…”

“…These specific symptoms on tobacco may in part explain the global average of Lin's concordance coefficients for the SAD-aided estimates in our study were not greater than 0.9. In fact, such level is not uncommon in SADs validation studies that used Lin's CCC for summarizing accuracy and precision (Yadav et al 2013;Braido et al 2014;Schwank and Del Ponte 2014;Dolinski et al 2017;Araújo et al 2019;Melo et al 2020, Sterling et al 2020. However, the reasons for not reaching overall means of Lin's CCC > 0.9 when using SADs can be of various natures and should be explored further.…”

“…It is expected that for raters whose estimates are innately very accurate, the SADs usually have little positive effect, if any. In some occasions, the use of the SADs can be detrimental as shown in regression plots between gain/loss from using the SADs and unaided estimates (González-Domínguez et al 2014;Franceschi et al 2020;Melo et al 2020). This kind of response is not understood or easy to explain as it may be due to random error or specific test conditions.…”

“…A multi-laboratory validation was performed to check whether the proposed method (SADs + instruction) leads to consistent and reproducible results across groups of users. Differently from the work by Melo et al (2020), we used the same set of reference images across the laboratories and provided standard instructions in a slide presentation. At each laboratory, one examiner was responsible for testing the raters and making the following decisions: (a) select the raters; (b) use the more convenient and available image projection setting (wall projection, computer screen); and (3) define the oral explanation to complement the instructions presented textually and graphically in a slide presentation.…”

“…Moreover, it is poorly known whether the validation results for one SADs are consistent across different groups where the testing conditions may not be the same. Recently, an observational study to validate a SAD set developed for Botrytis symptoms on Gerbera leaves suggested that differences in methodology, instruction, software, or rater group likely explained the inconsistent results between two laboratories (Melo et al 2020).…”