Outsourcing image search services to public clouds is an ever-increasing trend. However, directly outsourcing image datasets to untrusted clouds introduces privacy concerns. Several secure image retrieval schemes have been proposed recently. However, most of them require participation from image owners when building secure indexes, which wastes many computational resources of the image owners. Several schemes are proposed to solve this problem, but they suffer from low search accuracy on large datasets. In this paper, we propose the first secure image retrieval scheme that simultaneously solves these two problems. To obtain higher search accuracy, we extract image features via fine-tuned convolutional neural networks. Then, the image features are encrypted by using the secure k-Nearest Neighbor algorithm. To improve search speed and reduce the cost of image owners, we let cloud servers locally build a secure hierarchical index graph by using the encrypted image features. Besides, the secure index can be built and updated in parallel. We provide security analysis for the proposed scheme. Performance evaluations on the CIFAR-10 dataset show that the proposed scheme is practical. Moreover, compared with a recent scheme, our scheme can save more index construction time and cost of image owners when building secure indexes. INDEX TERMS Content-based image retrieval, encrypted image retrieval, navigable small world graph, secure index.