Using Generalizability Theory to Evaluate the Applicability of a Serial Bayes Model in Estimating the Positive Predictive Value of Multiple Psychological or Medical Tests

Abstract

Introduction: It is a common finding that despite high levels of specificity and sensitivity, many medical tests are not highly effective in diagnosing diseases exhibiting a low prevalence within a clinical population. What is not widely known or appreciated is how the results of retesting a patient using the same or a different medical or psychological test impacts the estimated probability that a patient has a particular disease. In the absence of a 'gold standard' special techniques are required to understand the error structure of a medical test. Generalizability can provide guidance as to whether a serial Bayes model accurately updates the positive predictive value of multiple test results. Methods: In order to understand how sources of error impact a test's outcome, test results should be sampled across the testing conditions that may contribute to error. A generalizability analysis of appropriately sampled test results should allow researchers to estimate the influence of each error source as a variance component. These results can then be used to determine whether, or under what conditions, the assumption of test independence can be approximately satisfied, and whether Bayes theorem accurately updates probabilities upon retesting. Results: Four hypothetical generalizability study outcomes are displayed as variance component patterns. Each pattern has a different practical implication related to achieving independence between test results and deriving an enhanced PPV through retesting an individual patient. Discussion: The techniques demonstrated in this article can play an important role in achieving an enhanced positive predictive value in medical and psychological diagnostic testing and can help ensure greater confidence in a wide range of testing contexts.