We derive constraints on two-dimensional conformal field theories with higher spin symmetry due to unitarity, modular invariance, and causality. We focus on CFTs with W N symmetry in the "irrational" regime, where c > N − 1 and the theories have an infinite number of higher-spin primaries. The most powerful constraints come from positivity of the Kac matrix, which (unlike the Virasoro case) is non-trivial even when c > N − 1. This places a lower bound on the dimension of any non-vacuum higher-spin primary state, which is linear in the central charge. At large c, this implies that the dual holographic theories of gravity in AdS 3 , if they exist, have no local, perturbative degrees of freedom in the semi-classical limit.