Objective. While electroencephalography (EEG)-based brain–computer interfaces (BCIs) have many potential clinical applications, their use is impeded by poor performance for many users. To improve BCI performance, either via enhanced signal processing or user training, it is critical to understand and describe each user’s ability to perform mental control tasks and produce discernible EEG patterns. While classification accuracy has predominantly been used to assess user performance, limitations and criticisms of this approach have emerged, thus prompting the need to develop novel user assessment approaches with greater descriptive capability. Here, we propose a combination of unsupervised clustering and Markov chain models to assess and describe user skill. Approach. Using unsupervised K-means clustering, we segmented the EEG signal space into regions representing pattern states that users could produce. A user’s movement through these pattern states while performing different tasks was modeled using Markov chains. Finally, using the steady-state distributions and entropy rates of the Markov chains, we proposed two metrics taskDistinct and relativeTaskInconsistency to assess, respectively, a user’s ability to (i) produce distinct task-specific patterns for each mental task and (ii) maintain consistent patterns during individual tasks. Main results. Analysis of data from 14 adolescents using a three-class BCI revealed significant correlations between the taskDistinct and relativeTaskInconsistency metrics and classification F1 score. Moreover, analysis of the pattern states and Markov chain models yielded descriptive information regarding user performance not immediately apparent from classification accuracy. Significance. Our proposed user assessment method can be used in concert with classifier-based analysis to further understand the extent to which users produce task-specific, time-evolving EEG patterns. In turn, this information could be used to enhance user training or classifier design.