Background After a stroke, experts recommend regular monitoring and kinematic assessments of patients to objectively measure motor recovery. With the rise of new technologies and increasing needs for neurorehabilitation, an interest in virtual reality has emerged. In this context, we have developed an immersive virtual reality version of the Box and Block Test (BBT-VR). The aim of this study was to assess the concurrent validity of the BBT-VR among patients with stroke and healthy participants. Methods Twenty-three healthy participants and 22 patients with stroke were asked to perform the classical Box and Block Test (BBT) and BBT-VR three times with both hands. Concurrent validity was assessed through correlations between these two tests and reliability of the BBT-VR through correlation on test–retest. Usability of the BBT-VR was also evaluated with the System Usability Scale. Hand kinematic data extracted from controller’s 3D position allowed to compute mean velocity (Vmean), peak velocity (Vpeak) and smoothness (SPARC). Results Results showed strong correlations between the number of blocks displaced with the BBT and the BBT-VR among patients with stroke for affected (r = 0.89; p < 0.001) and less-affected hands (r = 0.76; p < 0.001) and healthy participants for dominant (r = 0.58; p < 0.01) and non-dominant hands (r = 0.68; p < 0.001). Reliability for test–retest was excellent (ICC > 0.8; p < 0.001) and usability almost excellent (System Usability Scale = 79 ± 12.34%). On average participants moved between 30 and 40% less blocks during the BBT-VR than during the BBT. Healthy participants demonstrated significantly higher kinematic measures (Vmean = 0.22 ± 0.086 ms−1; Vpeak = 0.96 ± 0.341 ms−1; SPARC = − 3.31 ± 0.862) than patients with stroke (Vmean = 0.12 ± 0.052 ms−1; Vpeak = 0.60 ± 0.202 ms−1; SPARC = − 5.04[− 7.050 to − 3.682]). Conclusion The BBT-VR is a usable, valid and reliable test to assess manual dexterity, providing kinematic parameters, in a population of patients with stroke and healthy participants. Trial registration http://www.clinicaltrials.gov; Unique identifier: NCT04694833, Date of registration: 11/24/2020