Monte Carlo Simulation Studies

Spence, Ian

doi:10.1177/014662168300700403

Cited by 28 publications

(11 citation statements)

References 58 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More dimensions are needed to identify the content structure, and high-dimensional solutions are difficult to interpret, especially when substantial differences exist among the SMEs. To reduce the demands on the sl~Es, future research should explore incomplete MDS designs (e.g., Spence, 1982Spence, , 1983) and sorting procedures in which the SMEs are required to sort items into a limited number of categories according to their similarity. If SME congruence is not a concern, future research should consider averaging the similarity data over the SMEs to provide a single matrix for classical MDS analysis.…”

Section: Implications For Future Researchmentioning

confidence: 99%

Using Subject-Matter Experts to Assess Content Representation: An MDS Analysis

Sireci

Geisinger

1995

Applied Psychological Measurement

View full text Add to dashboard Cite

Demonstration of content domain representation is of central importance in test validation. An expanded version of the method of content evaluation proposed by Sireci & Geisinger (1992) was evaluated with respect to a national licensure examination and a nationally standardized social studies achievement test. Two groups of 15 subject-matter experts (SMEs) rated the similarity of all item pairs comprising a test, and then rated the relevance of the items to the content domains listed in the test blueprints. The similarity ratings were analyzed using multidimensional scaling (MDS); the item relevance ratings were analyzed using procedures proposed by Hambleton (1984) and Aiken (1980). The SMES' perceptions of the underlying content structures of the tests emerged in the MDS solutions. All dimensions were germane to the content domains measured by the tests. Some of these dimensions were consistent with the content structure specified in the test blueprint, others were not. Correlation and regression analyses of the MDS item coordinates and item relevance ratings indicated that using both item similarity and item relevance data provided greater information of content representation than did using either approach alone. The implications of the procedure for test validity are discussed and suggestions for future research are provided.

show abstract

Section: Implications For Future Researchmentioning

confidence: 99%

Using Subject-Matter Experts to Assess Content Representation: An MDS Analysis

Sireci

Geisinger

1995

Applied Psychological Measurement

View full text Add to dashboard Cite

show abstract

“…Some guidance can be obtained from monte carlo work (Spence, 1983;Spence & Graeff, 1974), mainly for the determination of optimal dimensionality. In comparing a constrained solution with an unconstrained one, the former will generally have worse fit.…”

Section: Goodness-of-fit and Stabilitymentioning

confidence: 99%

Constrained Multidimensional Scaling, Including Confirmation

Heiser

Meulman

1983

Applied Psychological Measurement

View full text Add to dashboard Cite

Constrained and confirmatory multidimensional scaling (MDS) are not equivalent. Constraints refer to the translation of either theoretical or data analytical objectives into computational specifications. Confirmation refers to a study of the balance between systematic and random variation in the data for modeling of the systematic part. Among the topics discussed from this perspective are the role of substantive theory in MDS studies, the type of constraints currently envisaged, and the relationships with other data analysis methods. This paper points out the possibility of using either sampling models or resampling schemes to study the stability of MDS solutions. Parallel to Akaike's (1974) information criterion for choosing one out of many models for the same data, a general stability criterion is proposed and illustrated, based on the ratio of within to total spread of configurations issued from resampling. Scaling Analysis and OptimizationThe coupling of constraints and scaling, although not new, is still extremely vital. The first modem scaling model, developed by Thurstone (e.g., 1927), immediately entailed the necessity of a judicious choice of constraints, as classified in the five famous 6bc~sese&dquo; These were internal constraints in the sense that the modified model would still accommodate the same set of observations, possibly less accurately, but with more predictive value. Guttman (1946) took a number of important additional steps. He showed that Thurstone's s method, relying on an underlying multivariate normal distribution, could be replaced by an approach in which the optimization of a goodness-of-fit function is the central concept. This was an extension of similar developments occurring at that time in test theory and in contingency table analysis (for current reviews, s~e l~lishisato, 1980, and Gin, 1981) to the scaling of paired comparison and rank order data. Moreover, Guttman showed that introducing an additional type of constraint was not a problem. The proper objective would be to find scale values for the stimulus objects that are &dquo;best able to reproduce the judgment of each person in the population on each comparison&dquo; (Guttman, 1946, p. 14~) and that are also perfectly linearly predictable from a predeterrrgined set of (design) variables.Although through hindsight it seems clear that it would have been possible to formulate a constrained multidimensional scaling (MDS) method within Guttman's framework, the early paradigm for MDS was to split the problem into two distinct steps. From the very first published psychometric application by Klingberg (1941)-based on earlier work by Richardson (1938)-to the authoritative presentation in Torgerson's (1958) book, it is evident that the main burden and first step of the analysis was to find a one-dimensional scale of psychological distances between pairs of stimulus objects. This was usually done by any of the varat UNIVERSITE DE MONTREAL on June 23, 2015 apm.sagepub.com Downloaded from

show abstract

“…We shall investigate, tentatively, if our leave-one-out method can be used to determine a "correct" or "optimal" dimensionality. Previous procedures for choosing dimensionality are based on large scale Monte Carlo studies, and were reviewed recently by Spence (1983). Monte Carlo studies of robustness of MDS and of dimension estimation differ from our scheme, because Monte Carlo methods are used in situations in which the "correct" answer to the question that is studied is known beforehand.…”

Section: Introductionmentioning

confidence: 99%