The interrater reliability of SAPS II and SAPS 3

Strand, Kristian; Strand, Liv Inger

doi:10.1007/s00134-010-1772-1

Cited by 25 publications

(15 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our results are comparable to those from Strand et al [7], who reported similar difficulties for Norway junior doctors in assessing heart rate and systolic blood pressure. A mathematical explanation of this problem could be that five (four) choices are given for scoring of systolic blood pressure (heart rate) whereas the rating of the other physiological variables is generally less demanding.…”

Section: Discussionsupporting

confidence: 90%

“…Adequate interrater reliability of SAPS II has been reported in few studies [7, 8] and small differences in values of some SAPS II variables between observers have determined important differences in scores [8]. The Acute Physiology and Chronic Health Evaluation II scoring system (APACHE II) [9] has been more extensively studied, and reliable overall APACHE II scores have been achieved by various healthcare workers (trained hospital abstractors, nurses, resident physicians, and intensivists) [10–16].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Critical Care Nurses Inadequately Assess SAPS II Scores of Very Ill Patients in Real Life

Perren

Previsdomini

Perren

et al. 2012

Critical Care Research and Practice

View full text Add to dashboard Cite

Background. Reliable ICU severity scores have been achieved by various healthcare workers but nothing is known regarding the accuracy in real life of severity scores registered by untrained nurses. Methods. In this retrospective multicentre audit, three reviewers independently reassessed 120 SAPS II scores. Correlation and agreement of the sum-scores/variables among reviewers and between nurses and the reviewers' gold standard were assessed globally and for tertiles. Bland and Altman (gold standard—nurses) of sum scores and regression of the difference were determined. A logistic regression model identifying risk factors for erroneous assessments was calculated. Results. Correlation for sum scores among reviewers was almost perfect (mean ICC = 0.985). The mean (±SD) nurse-registered SAPS II sum score was 40.3 ± 20.2 versus 44.2 ± 24.9 of the gold standard ( P < 0.002 for difference) with a lower ICC (0.81). Bland and Altman assay was +3.8 ± 27.0 with a significant regression between the difference and the gold standard, indicating overall an overestimation (underestimation) of lower (higher; >32 points) scores. The lowest agreement was found in high SAPS II tertiles for haemodynamics ( k = 0.45–0.51). Conclusions. In real life, nurse-registered SAPS II scores of very ill patients are inaccurate. Accuracy of scores was not associated with nurses' characteristics.

show abstract

Section: Discussionsupporting

confidence: 90%

Section: Introductionmentioning

confidence: 99%

Critical Care Nurses Inadequately Assess SAPS II Scores of Very Ill Patients in Real Life

Perren

Previsdomini

Perren

et al. 2012

Critical Care Research and Practice

View full text Add to dashboard Cite

show abstract

“…In these studies both SAPS II and SAPS 3 provided excellent discrimination [14-17, 21, 22, 25-27], but also overestimated hospital mortality [14,28] and poor calibration was usually found between them or even being poorer for SAPS 3 [15-17, 27, 28]. Like in the present study, both SAPS II and SAPS 3 have shown adequate inter-rater reliability, but the standardized mortality ratios are still likely to be influenced by the rater's scoring practice [29].…”

supporting

confidence: 52%

Calibration strategies to validate predictive models: is new always better?

Serrano

2012

Intensive Care Med

View full text Add to dashboard Cite

Calibration along with discrimination is an important measure of accuracy to validate predictive logistic regression models. Most predictive models in intensive care such as Simplified Acute Physiology Score (SAPS) II [1] and SAPS 3 [2,3] consider the binary outcome whether a patient will be alive or dead at hospital discharge. Discrimination measures how well the model can distinguish between patients who die and those who survive. Discrimination is usually assessed by the area under the receiver operating characteristic curve (AU-ROC) [4]. This statistic evaluates each pair of observations that have different outcomes and calculates the proportion of times when the patient who died had a higher predicted mortality than did the survivor. The AU-ROC ranges from 0.50 (no discrimination: complete binary random of 50 % similar to flipping a coin) to 1.00 (100 % correct discrimination of the model) [4].Calibration measures the model's ability to generate predictions that are on average close to the average observed outcome. Calibration has traditionally been approached in two steps. First, to investigate the overall ability to correctly relate the actual occurrence of the event to its estimated probability using statistical methods. The most widely used method is the HosmerLemeshow (H-L) test [5], which examines how well the percentage of observed deaths matches the percentage of predicted deaths over deciles of predicted risk. A p value greater than 0.05 is needed to conclude that there are no significant differences between the observed and expected outcomes and therefore the model has good overall calibration. Second, to localize possible deviations across risk strata by means of calibration plots with observed outcomes versus expected probabilities of mortality. The calibration plot, also named calibration curve, intends to provide complementary information over subsets. If the model calibrates well, there will not be a substantial deviation from the 45°line of perfect fit or bisector. On the contrary, miscalibration of the model will be a function of expected probability.The H-L test is easy to compute and its interpretation is intuitive, but it has acknowledged limitations such as being very sensitive to sample size [6][7][8]. The traditional plot or calibration curve also has some disadvantages: first, rather than a curve, it is a jagged line connecting the points in the plot; second, it is not accompanied by any information on the statistical significance of deviations from the bisector [9].Methods to measure the calibration of predictive models have remained unchanged for a long time, but new insights are now provided. In this issue of Intensive Care Medicine, Poole et al.[10] present a multicenter study aimed at comparing the performance of SAPS II and SAPS 3 in predicting hospital mortality. Discrimination was measured as usual using ROC curves and both models discriminated fairly. Interestingly, the calibration ability of the systems was measured using a new approach named calibration belt [9].Both syste...

show abstract

“…The reliability of data collection in SIR is not universally tested and this can be considered one of the weakest areas of SIR. The inter-rater reliability of SAPS3 coding has previously been shown to be adequate in Norway [74].…”

Section: Reliability Of Codingmentioning

confidence: 98%

The significance of risk adjustment for the assessment of results in intensive care. : An analysis of risk adjustment models used in Swedish intensive care.

Engerström¹

2018

Linköping University Medical Dissertations

View full text Add to dashboard Cite

The interrater reliability of SAPS II and SAPS 3

Cited by 25 publications

References 13 publications

Critical Care Nurses Inadequately Assess SAPS II Scores of Very Ill Patients in Real Life

Critical Care Nurses Inadequately Assess SAPS II Scores of Very Ill Patients in Real Life

Calibration strategies to validate predictive models: is new always better?

The significance of risk adjustment for the assessment of results in intensive care. : An analysis of risk adjustment models used in Swedish intensive care.

Contact Info

Product

Resources

About