Ordinal data in a repeated measures design of a crossover study for rare diseases usually do not allow for the use of standard parametric methods, and hence, nonparametric methods should be considered instead. However, only limited simulation studies in settings with small sample sizes exist. Therefore, starting from an Epidermolysis Bullosa simplex trial with the above‐mentioned design, a rank‐based approach using the R package nparLD and different generalized pairwise comparisons (GPC) methods were compared impartially in a simulation study. The results revealed that there was not one single best method for this particular design, because a trade‐off exists between achieving high power, accounting for period effects, and for missing data. Specifically, nparLD as well as the unmatched GPC approaches do not address crossover aspects, and the univariate GPC variants partly ignore the longitudinal information. The matched GPC approaches, on the other hand, take the crossover effect into account in the sense of incorporating the within‐subject association. Overall, the prioritized unmatched GPC method achieved the highest power in the simulation scenarios, although this may be due to the specified prioritization. The rank‐based approach yielded good power even at a sample size of , whereas the matched GPC method could not control the type I error.