Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment

Abstract

Quantitative risk assessments facilitate the decisions of risk managers. In the EU, risk assessment in food and feed safety is the responsibility of the European Food Safety Authority (EFSA). Quantitative risk models should be informed by systematically reviewed scientific evidence, however, in practice empirical evidence is often limited: in such cases it is necessary to turn to expert judgement. Psychological research has shown that unaided expert judgement of the quantities required for risk modelling -and particularly the uncertainty associated with such judgements -is often biased, thus limiting its value. Accordingly methods have been developed for eliciting knowledge from experts in as unbiased a manner as possible. In 2012, a working group was established to develop guidance on expert knowledge elicitation appropriate to EFSA's remit. The resulting Guidance first presents expert knowledge elicitation as a process beginning with defining the risk assessment problem, moving through preparation for elicitation (e.g. selecting the experts and the method to be used) and the elicitation itself, culminating in documentation. Those responsible for managing each of these phases are identified. Next three detailed protocols for expert knowledge elicitation are given -that can be applied to real-life questions in food and feed safety -and the pros and cons of each of these protocols are examined. This is followed by principles for overcoming the major challenges to expert knowledge elicitation: framing the question; selecting the experts; eliciting uncertainty; aggregating the results of multiple experts; and documenting the process. The results of a web search on existing guidance documents on expert elicitation are then reported, along with case studies illustrating some of the protocols of the Guidance. Finally, recommendations are made in the areas of training, organisational changes, expert identification and management, and further developments of expert knowledge elicitation methodology within EFSA. EXECUTIVE SUMMARYIn the European Union, risk assessment in food and feed safety is the responsibility of the European Food Safety Authority (EFSA) -the outcomes of EFSA's risk assessments facilitate the decisions of risk managers. The use of comprehensive quantitative models for risk assessment is well established, and the approach will be increasingly adopted by EFSA in the future. These models should be informed by systematically reviewed scientific evidence. However, empirical evidence is often limited, absent, conflicting, of doubtful relevance, open to alternative interpretations or not accessible via publicly available information sources (e.g. bibliographic databases, scientific journals, or websites). In such cases, reliable information can be obtained from knowledgeable experts using systematic and standardized methods for eliciting their knowledge. Even where there is sufficient empirical evidence, consultation with relevant experts can be beneficial, for instance, at the start of a proces...

Quantifying uncertainty about future antimicrobial resistance: Comparing structured expert judgment and statistical forecasting methods

“…Experts have relevant knowledge about these issues and their impact on future resistance rates, which should be included in forecasts. Expert forecasts can be limited by heuristics and biases, but structured elicitation approaches, such as the Classical Model, are a way to minimize their impact on results [41].…”

“…Experts can also directly supply forecasted values, a technique that is especially useful when historical data are not available or have limited predictive value [39]. Expert judgment can be subject to a range of biases [40], but a set of techniques called “expert elicitation” enable experts to provide estimates and quantify their uncertainty about parameters of interest—including projecting future events and values—in a way that minimizes opportunities for bias [38,41,42]. One such method, the Classical Model of structured expert judgment, has been used in over 80 applications, including estimating the burden of foodborne disease [43–45] and forecasting volcanic activity on the island of Montserrat [46–48], the introduction of invasive species in the Great Lakes [49], and future sea level rise caused by ice sheet melt [50].…”

Risk Benefit Assessment of foods: Key findings from an international workshop

“…the metadata, are needed to fill these data gaps for RBA. If observational or experimental data are lacking, another option is to gather information through expert elicitation (Cooke, 1991;EFSA, 2014a;EPA, 2009). This technique is already used in microbial risk assessment (Albert et al, 2012;Pujol, Johnson, Magras, Albert, & Membré, 2015; Van der Fels-Klerx, Cooke, Nauta, Goossens, & Havelaar, 2005) and more generally in food safety (Hald et al, 2016).…”

Slaughter of Pregnant Cattle in Denmark: Prevalence, Gestational Age, and Reasons

“…However, no recent data are reported, and the two studies mentioned above focused only on dairy cattle rather than cattle kept in an extensive production system, which has a different production cycle. A recent systematic literature review followed by expert knowledge elicitation [21] characterised the main reasons for slaughter of pregnant cattle as economic (e.g., low productivity), management-related (e.g., false-negative pregnancy diagnoses or pregnant cows being calmer than non-pregnant cows and thus resulting in fewer injuries in the herd), and health and welfare related (e.g., lameness, mastitis, disease control) [10].…”