Fabric classification plays a crucial role in the modern textile industry and fashion market. In the early stage, traditional neural network methods were adopted to identify fabrics with the drawback of restricted fabric type and poor accuracy. Combining multi-frame temporality and analysing fabric graph data made from fabric motion features, this paper proposes a novel hybrid model that introduces the concept of graph networks to classify 30 textile materials in a public database. We utilize the graph inductive representation learning method (GraphSAGE, Graph Sample and Aggregate) to extract node embedding features of the fabric. Moreover, bidirectional gated recurrent unit and layer attention mechanism (BiGRU-attention) are employed in the last layer of aggregation to calculate the score of previous cells. Intending to further enhance performance, we link the jump connection with adaptive selection aggregation frameworks to determine the influential region of each node. Our method breaks through the limitation that the original methods can only classify a few fabrics with great classification results.