Estimating coefficients in linear models: It don't make no nevermind.

Wainer, Howard

doi:10.1037/0033-2909.83.2.213

Cited by 623 publications

(367 citation statements)

References 7 publications

Supporting

Mentioning

351

Contrasting

Unclassified

Order By: Relevance

“…The summary statistics for these three models are very similar, illustrating that the multiple regression model is not sensitive to small variations in the weights chosen (cf. Wainer, 1976). In fact, the regression model with the weights set by the CAC has the best correlation with human scores for sets of three tasks.…”

Section: Resultsmentioning

confidence: 91%

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

This report presents the results of a research and development effort for SpeechRater SM Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL ® ) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short-and long-term goals for iterative improvements to SpeechRater scoring. which is used by prospective test takers to prepare for the official TOEFL iBT test. This study reports the development and validation of the system for low-stakes practice purposes. The process we followed to build this system represented a principled approach to maximizing 2 essential qualities: substantively meaningful and technically sound. In developing and evaluating the features and the scoring models to predict human assigned scores, we engaged both content and technical experts actively to ensure the construct representation and technical soundness of the system. We compared primarily two alternative methodologies of building scoring modelsmultiple regression and classification trees-in terms of their construct representation and empirical performance in predicting human scores. Based on the evaluation results, we concluded that a multiple regression model with feature weights determined by content experts was superior to the other competing models evaluated.We then used an argument-based approach to integrate and evaluate the existing evidence to support the use of SpeechRater SM v1.0 in a low-stakes practice environment. The argumentbased approach provided a mechanism for us to articulate the strengths and weaknesses in the validity argument for using SpeechRater v1.0 and put forward a transparent argument for using it for a low-stakes practice environment. In particular, the construct representation of the multiple regression model with expert weights was sufficiently broad to justify its use in a low-stakes application. While some higher-order aspects of the speaking construct (such as content and organization) are missing, more basic aspects of the construct (such as pronunciation and fluency) are richly represented. In addition, these different parts of the speaking construct tend to be highly correlated, so that the absence of higher order factors is not as detrimental to the model's agreement with human raters as it otherwise might be. The model's agreement with human raters was not sufficiently high to support high-stakes decisions but was still suitable for use in low-stakes applications. The correlation of the 6-item aggregate score with human raters was .57 and was deemed acceptable given the lo...

show abstract

Section: Resultsmentioning

confidence: 91%

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…The ways in which outputs are relatively insensitive to changes in coefficients (provided changes in sign are not involved) have been investigated most recently by Green (1977), Because each of the random models is positively correlated with the criterion, the correlation of their average, which is the unit-weighted model, is higher than the average of the correlations. Wainer (1976), Wainer and Thissen (1976), W. M. Edwards (1978), and Gardiner and Edwards (197S). Dawes and Corrigan (1974, p. 105) concluded that "the whole trick is to know what variables to look at and then know how to add."…”

Section: Paul Meehps (1954) Book Clinical Versus Statistical Predictimentioning

confidence: 99%

The robust beauty of improper linear models in decision making.

Dawes¹

1979

American Psychologist

1,713

763

View full text Add to dashboard Cite

Proper linear models are those in which predictor variables are given weights in such a way that the resulting linear composite optimally predicts some criterion of interest; examples of proper linear models are standard regression analysis, discriminant function analysis, and ridge regression analysis. Research summarized in Paul Meehl's book on clinical versus statistical prediction-and a plethora of research stimulated in part by that book-all indicates that when a numerical criterion variable (e.g., graduate grade point average) is to be predicted from numerical predictor variables, proper linear models outperform clinical intuition. Improper linear models are those in which the weights of the predictor variables are obtained by some nonoptimal method; for example, they may be obtained on the basis of intuition, derived from simulating a clinical judge's predictions, or set to be equal. This article presents evidence that even such improper linear models are superior to clinical intuition when predicting a numerical criterion from numerical predictors. In fact, unit (i.e., equal) weighting is quite robust for making such predictions. The article discusses, in some detail, the application of unit weights to decide what bullet the Denver Police Department should use. Finally, the article considers commonly raised technical, psychological, and ethical resistances to using linear models to make important social decisions and presents arguments that could weaken these resistances.

show abstract

“…The fact that linear (or almost linear) regression models human decision-making quite well is also well known. It was reported by decisionmaking researchers during the Seventies (Dawes, 1974;Wainer, 1976). Furthermore, machine learning models have been often shown to be more accurate then well established statistical techniques (Tarn, 1992;Thrun et al, 1991), in particular, when compared with various regression models.…”

Section: Discussionmentioning

confidence: 99%