SummaryMobile sleep‐monitoring devices for consumer use have been gaining traction as a possible replacement to traditional polysomnography recordings. Such devices potentially offer detailed sleep analysis without requiring the use of designated sleep labs operated by qualified technicians. However, the accuracy of these mobile devices is often not sufficiently evaluated by independent researchers. Here, we compared the performance of two popular mobile electroencephalogram‐based systems, the DREEM 3 headband and the Zmachine Insight+. Both devices can be used by participants with minimal training, and provide detailed sleep scoring previously validated by the respective developers in comparison to the gold‐standard of polysomnography. A total of 25 participants used both devices simultaneously to record their sleep for two consecutive nights while also keeping a sleep log. We compared the devices' performance, both with each other and in relation to the sleep logs, using several well‐known sleep metrics. In addition, we developed a Bayesian lower limit for the devices' expected epoch‐by‐epoch sleep stage agreement based on their previously published agreement with polysomnography, and compared it with our empirical findings. Results suggest that the Zmachine tends to overestimate periods of wakefulness, likely at the expense of N1/N2 detection, whereas the DREEM tends to underestimate wakefulness and mistake it for N1/N2, with both results more pronounced than previously reported. In addition, we found that the agreement between the devices tends to increase from night 1 to night 2. We formulate several recommendations for how best to use these devices based on our results.