Collinear and noncollinear spin structures of wurtzite phase CoO often appearing in nanosized samples are investigated using first-principles density functional theory calculations. We examined the total energy of several different spin configurations, electronic structure, and the effective magnetic coupling strengths. It is shown that the AF3-type antiferromagnetic ordering is energetically most stable among possible collinear configurations. Further, we found that a spiral spin order can be stabilized by including the relativistic spin-orbit coupling and the noncollinearity of spin direction. Our result suggests that a noncollinear spin ground state can be observed in the transition-metal-oxide nanostructures, which adds an interesting aspect to the nanomagnetism study.