We investigated the influence of spins on surface of nanodiamonds (NDs) to the longitudinal relaxation time (T1) and transverse relaxation time (T2) of nitrogen vacancy (NV) centers in ND. A spherical model of the NDs was suggested to account for the experimental results of T1 and T2, and the density of surface spins was roughly estimated based on the statistical analysis of experimental results of 72 NDs containing a single NV center. For NDs studied here, the T1 of NV center inside is highly dependent to the surface spins of the NDs. However, for the T2 of NV center, intrinsic contributions must be much pronounced than that by surface spins. In other words, T1 of an NV center in NDs is more sensitive to the change of the surface spin density than T2.