Psychometric modeling of decision making via game play

Abstract

Abstract-We build a model for the kind of decision making involved in games of strategy such as chess, making it abstract enough to remove essentially all game-specific contingency, and compare it to known psychometric models of test taking, item response, and performance assessment. Decisions are modeled in terms of fallible agents Z faced with possible actions ai whose utilities ui = u(ai) are not fully apparent. The three main goals of the model are prediction, meaning to infer probabilities pi for Z to choose ai; intrinsic rating, meaning to assess the skill of a person's actual choices ai t over various test items t; and simulation of the distribution of choices by an agent with a specified skill set. We describe and train the model on large data from chess tournament games of different ranks of players, and exemplify its accuracy by applying it to give intrinsic ratings for world championship matches.

Assessing Human Error Against a Benchmark of Perfection

“…This makes it possible to analyze the moves of strong human players, in a large-scale fashion, comparing their choices to those of an engine. This has been pursued very effectively in the last several years by Biswas and Regan [2,3,23]; they have used the approach to derive interesting insights including proposals for how to estimate the depth at which human players are analyzing a position.…”

Six Suggestions for Research on Games in Cognitive Science

“…They began by showing that the skill levels of individual players can be estimated accurately by examining how their individual move choices correlate with those of top chess engines: Stronger players more often play one of the computer's preferred moves, which is taken to mean that the stronger players are playing better moves (Regan & Haworth, ; Regan & Biswas, ). Interestingly, the correspondence between individual move quality and rating has been consistent over time, meaning that the top players of the current era (who are rated in the 2750–2850 range on the Elo scale) are playing better chess than the top players of earlier eras (who were rated 2600–2750).…”

A Comparative Review of Skill Assessment: Performance, Prediction and Profiling

“…Table 1 highlights some games analysed with STOCKFISH 3.0. [14,22,23] identifies the BP i which best fits the observed play: it is essentially frequentist. The probability of BP(c i ) playing moves m 1 -m k is p(c i )  q j,i and c i is found to maximize p(c i ).…”