Performance and Prediction: Bayesian Modelling of Fallible Choice in Chess

Abstract

Abstract. Evaluating agents in decision-making applications requires assessing their skill and predicting their behaviour. Both are well developed in Poker-like situations, but less so in more complex game and model domains. This paper addresses both tasks by using Bayesian inference in a benchmark space of reference agents. The concepts are explained and demonstrated using the game of chess but the model applies generically to any domain with quantifiable options and fallible choice. Demonstration applications address questions frequently asked by the chess community regarding the stability of the rating scale, the comparison of players of different eras and/or leagues, and controversial incidents possibly involving fraud. The last include alleged under-performance, fabrication of tournament results, and clandestine use of computer advice during competition. Beyond the model world of games, the aim is to improve fallible human performance in complex, high-value tasks.

Understanding Distributions of Chess Performances

“…Haworth [1,2] and with DiFatta and Regan [3,12,4] developed models of fallible decision agents that can be trained on players' games and calibrated to a wide range of skill levels. Their main difference from [5][6][7] is the use of Multi-PV analysis to obtain uthoritative values for all reasonable options, not just the top move(s) and the move played.…”

“…However, we argue that for an intrinsic standard of quality by which to judge possible rating drift, one needs greater depth, the full move context, and a variety of scientific approaches. The papers [3,12] apply Bayesian analysis to characterize the performance of human players using a spectrum of reference fallible agents. The work reported in [4] and this paper uses a method patterned on multinomial Bernoulli trials, and obtains a corresponding spectrum.…”

A Comparative Review of Skill Assessment: Performance, Prediction and Profiling

“…An important advantage is that his simple formula can be used iteratively as each new observation arrives. Skilloscopy is the name given to the assessment of skill by Bayesian Inference [2][3][4][5][6][7][8][9]. It proceeds from initial inherited or presumed probabilities p i that AP 'is' BP(c i ): AP's initial presumed apparent competence ac is therefore  i p i .…”

“…The Reference ELO Player RP e , a Reference Player with ELO e may be defined by analyzing the moves of a set of players with ELO e, e.g., [2690,2710]. This was done [7][8][9], in fact restricting chosen games to those between two such players. 1 The players' ratings in MM/EV, AD/ASD and SR/SK terms may be used to calibrate their respective scales.…”

“…Bayesian inference as used by Haworth et al [2][3][4][5][6][7][8][9] is the natural vehicle for this as it combines an expression of prior belief with the modification of that belief in the light of subsequent evidence. Players may also be profiled from a 'cold start' if 'know nothing' priors are adopted.…”