Validity: on the meaningful interpretation of assessment data

Downing, Steven M.

doi:10.1046/j.1365-2923.2003.01594.x

Cited by 1,195 publications

(1,036 citation statements)

References 31 publications

Supporting

Mentioning

955

Contrasting

Unclassified

Order By: Relevance

“…These behaviors alone do not constitute teamwork; however, they certainly may influence team interactions. [231][232][233] Finally, we used established frameworks for abstracting tool validity 32,33 and study quality; 36 however, these frameworks do not encompass every aspect of validity and/or quality present in studies.…”

Section: Discussionmentioning

confidence: 99%

“…We used an established framework to categorize the validity of instruments [32][33][34] that has been used in similar evaluations of assessment tools. 26 This framework includes five categories of validity evidence: 1) content (the degree to which the tool content reflects the construct being measured); 2) response process (training of raters to use the tool); 3) internal structure (instrument reliability including internal consistency, interrater, intrarater, and test-retest reliability); 4) relationships to other variables (relationship between scores and other variables measuring the same construct); and 5) consequences (outcomes associated with tool scores).…”

Section: Data Extractionmentioning

confidence: 99%

See 1 more Smart Citation

Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes

et al. 2013

View full text Add to dashboard Cite

OBJECTIVE: Valid teamwork assessment is imperative to determine physician competency and optimize patient outcomes. We systematically reviewed published instruments assessing teamwork in undergraduate, graduate, and continuing medical education in general internal medicine and all medical subspecialties. DATA SOURCES: We searched MEDLINE, MEDLINE In-process, CINAHL and PsycINFO from January 1979 through October 2012, references of included articles, and abstracts from four professional meetings. Two content experts were queried for additional studies. STUDY ELIGIBILITY: Included studies described quantitative tools measuring teamwork among medical students, residents, fellows, and practicing physicians on single or multi-professional (interprofessional) teams. STUDY APPRAISAL AND SYNTHESIS METHODS:Instrument validity and study quality were extracted using established frameworks with existing validity evidence. Two authors independently abstracted 30 % of articles and agreement was calculated. RESULTS: Of 12,922 citations, 178 articles describing 73 unique teamwork assessment tools met inclusion criteria. Interrater agreement was intraclass correlation coefficient 0.73 (95 % CI 0.63-0.81). Studies involved practicing physicians (142, 80 %), residents/fellows (70, 39 %), and medical students (11, 6 %). The majority (152, 85 %) assessed interprofessional teams. Studies were conducted in inpatient (77, 43 %), outpatient (42, 24 %), simulation (37, 21 %), and classroom (13, 7 %) settings. Validity evidence for the 73 tools included content (54, 74 %), internal structure (51, 70 %), relationships to other variables (25, 34 %), and response process (12, 16 %). Attitudes and opinions were the most frequently assessed outcomes. Relationships between teamwork scores and patient outcomes were directly examined for 13 (18 %) of tools. Scores from the Safety Attitudes Questionnaire and Team Climate Inventory have substantial validity evidence and have been associated with improved patient outcomes. LIMITATIONS: Review is limited to quantitative assessments of teamwork in internal medicine. CONCLUSIONS:There is strong validity evidence for several published tools assessing teamwork in internal medicine. However, few teamwork assessments have been directly linked to patient outcomes. 3,4 Recognizing this, the Institute of Medicine, the Joint Commission, the Agency for Healthcare Research and Quality (AHRQ), and others have made teamwork a top priority in their recommendations for improving healthcare. [5][6][7][8][9] Teamwork is also prominently positioned within the American Board of Internal Medicine (ABIM) requirements for maintenance of certification for internists, 10 as well as the Accreditation Council for Graduate Medical Education's core competencies, 11 milestones, 12 and medical student competencies. 13 As such, every physician at the undergraduate, graduate, and continuing professional level must demonstrate competency in teamwork.While there is broad agreement on the imperative to improve teamwork, there is littl...

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Data Extractionmentioning

confidence: 99%

Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes

et al. 2013

View full text Add to dashboard Cite

show abstract

“…This systematic review was performed to investigate in which way validity of portfolio data in medical education are studied and reported in scientific literature. The validity framework of Messick was used to investigate the various aspects of validity [2]. We aimed to formulate guidelines for future validity research of portfolios in the field of medical education.…”

mentioning

confidence: 99%

“…A BEME data coding form was used to collect data concerning context-and portfolio-related characteristics, study design and -quality, and the results of the validity studies. Next, the studies were classified according to the validity framework used by Messick [2] based on the methodology used. As such, different sources to prove evidence of validity could be disentangled.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Rectification: Best abstracts of the NVMO conference

Michels¹

2013

Perspect Med Educ

View full text Add to dashboard Cite

In the previous edition of Perspectives on Medical Education the award winning and nominated abstracts of the yearly conference of the Netherlands Association of Medical Education were presented. We regret that one of the nominated abstracts was left out by mistake. The missing abstract by Nele Michels is presented below. Bohn Stafleu van LoghumNominated Abstract A systematic review of validity issues related to portfolio use: hit the target and don't miss the point. Michels NRM 1 , Remmen R 1 , Denekens J 1 , van Rossum H 2 , de Winter BY 1 1 University of Antwerp, Antwerp, Belgium, 2 VU University Medical Center, Amsterdam, the Netherlands Background: Portfolios are used as instruments to support learning, competence achievement, and reflection [1]. When portfolios are used for assessment purposes, validity of portfolio data is paramount. This systematic review was performed to investigate in which way validity of portfolio data in medical education are studied and reported in scientific literature. The validity framework of Messick was used to investigate the various aspects of validity [2]. We aimed to formulate guidelines for future validity research of portfolios in the field of medical education.

show abstract

A Unified Approach to Validation, Reliability, and Education Study Design for Surgical Technical Skills Training

Sweet¹

2010

Arch Surg

View full text Add to dashboard Cite

While validation of the individual simulation tools is important, it is only one piece of a multimodal curriculum that in and of itself deserves examination and study. We propose concurrent validation throughout the design of simulation-based curriculum rather than once it is complete. We embrace the concept that validity and curriculum development are interdependent, ongoing processes that are never truly complete. Individual predictive, construct, content, and face validity aspects should not be considered separately but as interdependent and complementary toward an end application. Such an approach could help guide our acceptance and appropriate application of these exciting new training and assessment tools for technical skills training in medicine.

show abstract

Validity: on the meaningful interpretation of assessment data

Cited by 1,195 publications

References 31 publications

Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes

Teamwork Assessment in Internal Medicine: A Systematic Review of Validity Evidence and Outcomes

Rectification: Best abstracts of the NVMO conference

A Unified Approach to Validation, Reliability, and Education Study Design for Surgical Technical Skills Training

Contact Info

Product

Resources

About