Clouds play a significant role in weather forecasts, water cycle, and climate system. However, existing methods ignore the relations of ground‐based cloud images. In this letter, we propose a novel method named task‐based graph convolutional network (TGCN) for ground‐based cloud classification, which takes image relations into consideration. To this end, we construct the graph using convolutional neural network‐based features of ground‐based cloud images which are learned in a supervised manner, and incorporate the graph computation into TGCN. Given that existing ground‐based cloud databases are with limited labeled training images and categorized according to different classification criteria, we release the largest ground‐based remote sensing cloud database (GRSCD) to provide a comparative study for different methods and to further improve the study of regional sky conditions. The experimental results on GRSCD manifest the effectiveness of TGCN for ground‐based cloud classification.