Summary Proposed is a discriminative feature modeling technique in three orthogonal planes (TOP) for human action recognition (HAR). Pyramidal histogram of orientation gradient‐TOP (PHOG‐TOP) and dense optical flow‐TOP (DOF‐TOP) techniques are utilized for salient motion estimation and description to represent the human action in a compact but distinct manner. The contribution of the work is to explicitly learn the gradual change of visual patterns using fusion of PHOG‐TOP and DOF‐TOP technique to discover the nature of the action. With this feature representation, dimensionality reduction is achieved by deep stacked autoencoders. The encoded feature representation is used in long short term memory (LSTM) classification for HAR. Experiments with better recognition rate demonstrate the discriminative power of the proposed descriptor. Moreover, the proposed modeling and LSTM classification outperforms the state of art methods for HAR.