The Kitaev model is a beautiful example of frustrated interactions giving rise to novel and unexpected physics. In particular, its classical version has remarkable properties stemming from exponentially large ground state degeneracy. Here, we present a study of magnetic clusters with spin-S moments coupled by Kitaev interactions. We focus on two cluster geometries -the Kitaev square and the Kitaev tetrahedron -that allow us to explicitly enumerate all classical ground states. In both cases, the space of ground states is large and self-intersecting, with non-manifold character. In the semi-classical large-S limit, we argue for effective low energy descriptions in terms of a single particle moving on non-manifold spaces. Remarkably, at low energies, the particle is tied down in bound states formed around singularities at self-intersection points. In the language of spins, the low energy physics is determined by a distinct set of states that lies well below other eigenstates. This set consists of admixtures of classical 'cartesian' states, with one state for each cartesian state in the classical ground state space. This brings out a strong role for cartesian states, a special class of classical ground states that are derived from dimer covers of the underlying lattice. This constitutes an example of order by singularity -where certain states from the classical ground state space are selected, not by fluctuations, but rather by singularities in the space.