Design and Testing of a Tool for Evaluating the Quality of Diabetes Consumer-Information Web Sites

Abstract

BackgroundMost existing tools for measuring the quality of Internet health information focus almost exclusively on structural criteria or other proxies for quality information rather than evaluating actual accuracy and comprehensiveness.ObjectiveThis research sought to develop a new performance-measurement tool for evaluating the quality of Internet health information, test the validity and reliability of the tool, and assess the variability in diabetes Web site quality.MethodsAn objective, systematic tool was developed to evaluate Internet diabetes information based on a quality-of-care measurement framework. The principal investigator developed an abstraction tool and trained an external reviewer on its use. The tool included 7 structural measures and 34 performance measures created by using evidence-based practice guidelines and experts' judgments of accuracy and comprehensiveness.ResultsSubstantial variation existed in all categories, with overall scores following a normal distribution and ranging from 15% to 95% (mean was 50% and median was 51%). Lin's concordance correlation coefficient to assess agreement between raters produced a rho of 0.761 (Pearson's r of 0.769), suggesting moderate to high agreement. The average agreement between raters for the performance measures was 0.80.ConclusionsDiabetes Web site quality varies widely. Alpha testing of this new tool suggests that it could become a reliable and valid method for evaluating the quality of Internet health sites. Such an instrument could help lay people distinguish between beneficial and misleading information.

Optimization and field use of a bioassay to monitor sea lice Lepeophtheirus salmonis sensitivity to emamectin benzoate

“…Inconsistencies between the outcomes of diagnostic tests run by different laboratories on the same sample using the same test have been shown to be partly due to subjectivity of the diagnostic test evaluations experienced by independent raters (see for example McClure et al 2005). Training and experience would likely improve inter-rater reliability (Seidman et al 2003, Sevatdal 2005. The inter-rater reliability of the current bioassay protocol was evaluated by comparing 2 independent rater's evaluations of sea lice responsiveness to EMB using the CCC.…”

Stroke Education Materials on the World Wide Web: An Evaluation of Their Quality and Suitability

“…13 Kim et al 5 reviewed 29 published rating tools and identified several key criteria for the evaluation of websites (see Table 1). Self-policing approaches, such as that established by Health on the Net (HON), enable websites to exhibit the HON code if they conform with a set of principles similar to those identified by Kim and colleagues.…”

Evaluating the Quality of Websites Offering Information on Female Hypoactive Sexual Desire Disorder

“…Similar to the Seidman diabetes website tool, which demonstrated moderate-to-high reliability, the modified HSDD website tool demonstrated moderate reliability (Seidman, Steinwachs, & Rubin, 2003). In contrast to the Seidman study which used an interrater methodology, this study used an intrarater methodology to assess reliability.…”

“…Indeed, the correlation between such criteria and the scientific reliability of Internet health information has been questioned (Ekman, Hall, & Litton, 2005). Seidman and colleagues have emphasized the need for tools with criteria that directly assess the scientific validity of health information present on a website (Seidman, Steinwachs, & Rubin, 2003). Rather than just assessing structural criteria, their tool evaluated criteria addressing the actual accuracy and comprehensiveness of diabetes information on websites.…”