Human action recognition is a computer vision challenge that involves identifying and classifying human movements and activities. The behavior of humans comprises movements of multiple body parts, and Graph Convolutional Networks (GCNs) have emerged as a promising approach for this task. However, most contemporary GCN methods perform graph convolution on the entire skeleton graph without considering that the human body consists of distinct body parts. To address these shortcomings, we propose a novel method that optimizes the representation of the skeleton graph by designing temporal and spatial convolutional blocks while introducing the Part-wise Adaptive Topology Graph Convolution (PAT-GC) technique. PAT-GC adaptively learns the segmentation of different body parts and dynamically integrates the spatial relevance between them. Furthermore, we utilize hierarchical modeling to divide the skeleton graph, capturing a more comprehensive representation of the human body. We evaluate our approach on three publicly available large datasets: NTU RGB + D 60, NTU RGB + D 120, and Kinetics Skeleton 400. Our experimental results demonstrate that our approach achieves state-of-the-art performance, thus validating the efficiency of our proposed technique for human action recognition.