General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal ? Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

To adequately assess students' competences, students are asked to provide proof of a performance. Ideally, open and real-life tasks are used for such performance assessment. However, to augment the reliability of the scores resulting from performance assessment, assessments are mostly standardised. This hampers the validity of the performance assessment. Comparative judgement (CJ) is introduced as an alternative judging method that does not require standardisation of tasks. The CJ method is based on the assumption that people are able to compare two performances more easily and reliable than assigning a score to a single one. This chapter provides insight in the method and elaborates on why this method is promising to generate valid, reliable measures in an efficient way, especially for large-scale summative assessments. Thereby, this chapter brings together the research already conducted in this new assessment domain.

Comparative judgment (CJ) is an alternative method for assessing competences based on Thurstone's law of comparative judgment. Assessors are asked to compare pairs of students work (representations) and judge which one is better on a certain competence. These judgments are analyzed using the Bradly-Terry-Luce model resulting in logit estimates for the representations. In this context, the Scale Separation Reliability (SSR), coming from Rasch modeling, is typically used as reliability measure. But, to the knowledge of the authors, it has never been systematically investigated if the meaning of the SSR can be transferred from Rasch to CJ. As the meaning of the reliability is an important question for both assessment theory and practice, the current study looks into this. A meta-analysis is performed on 26 CJ assessments. For every assessment, split-halves are performed based on assessor. The rank orders of the whole assessment and the halves are correlated and compared with SSR values using Bland-Altman plots. The correlation between the halves of an assessment was compared with the SSR of the whole assessment showing that the SSR is a good measure for split-half reliability. Comparing the SSR of one of the halves with the correlation between the two respective halves showed that the SSR can also be interpreted as an interrater correlation. Regarding SSR as expressing a correlation with the truth, the results are mixed.

Under normal circumstances, perception runs very fast and seemingly automatic. In just a few ms, we go from sensory features to perceiving objects. This fast time course does not only apply to general perceptual aspects but also to what we call higher-level judgements. Inspired by the study on 'very first impressions' by Bar, Neta, and Linz (2006, Emotion, 6, 269) the current research examined the speed and time course of three aspects of the aesthetic experience, namely beauty, specialness, and impressiveness. Participants were presented with 54 reproductions of paintings that covered a wide variety of artistic styles and contents. Presentation times were 10, 50, 100 and 500 ms in Experiment 1 and 20, 30 and 40 ms in Experiment 2. Our results not only show that consistent aesthetic judgements can be formed based on very brief glances of information, but that this speed of aesthetic impression formation also differs between different aesthetic judgements. Apparently, impressiveness judgements require longer exposure times than impressions of beauty or specialness. The results provide important evidence for our understanding of the time course of aesthetic experiences.From the moment we open our eyes, we clearly see the world around us and quickly extract information and meaning from it. Studies have found that when presented with real-world images, people are able to detect objects based on presentations as short as 50 ms and to recognize objects after only about 100 ms of presentation (see, e.g., Fei-Fei, Iyer, Koch, & Perona, 2007;Grill-Spector & Kanwisher, 2005). However, this fast processing comprises different stages. For example, Fei-Fei et al. (2007) found that before 40 ms, sensory-related features (light-dark) are dominant for people's perception of stimuli, while from around 50 ms of presentation, a shift to more object-related features occurs. Subjects are able to name very general object categories like manmade objects ('furniture', 'desk') and gradually get more detailed and accurate over time (Fei-Fei et al., 2007). Such a coarse-to-fine development of percepts has been found in many psychophysical studies (for a review, see Hegde, 2008) and is a core characteristic of the percept formation or microgenesis (Bachmann, 2000). What subjects perceive increases in detail the longer they see it, even if the image does not change. This happens on a very fast timescale (Fei-Fei et al., 2007;Hegde, 2008). A very vivid illustration that such speed and differentiated time course do not pertain only to general perceptual and identification aspects, such as colour or object category, but also to judgemental aspects *Correspondence should be addressed to M. Dorothee Augustin, Laboratory of Experimental Psychology, KU Leuven, Tiensestraat 102, box 3711, 3000 Leuven, Belgium (email: mdorothee.augustin@posteo.de). San Verhavert has moved since this study was undertaken and is now based at Department of Training and Educational Sciences, Research group Edubron, University of Antwerp, Gratiekapelstraat 10, 2000 Antwe...

Several studies have proven that comparative judgment (CJ) is a reliable and valid assessment method for a variety of competences, expert assessment, and peer assessment, and CJ is emerging as a possible approach to help maintain standards over time. For consecutive pairs of student works (representations) assessors are asked to judge which representation is better. It has been shown that random construction of pairs leads to very inefficient assessments, requiring a lot of pairwise comparisons to reach reliable results. Some adaptive selection algorithms using information from previous comparisons were proposed to increase the efficiency of CJ. These adaptive algorithms appear however to artificially inflate the reliability of CJ results through increasing the spread of the results. The current article proposes a new adaptive selection algorithm using a previously calibrated reference set. Using a reference set should eliminate the reliability inflation. In a real assessment, using reference sets of different reliability, and in a simulation study, it is proven that this adaptive selection algorithm is more efficient without reducing the accuracy of the results and without increasing the standard deviation of the assessment results. As a consequence, a reference-based adaptive selection algorithm produces high and correct reliability values in an efficient manner.