Robust Action Recognition under multiple views has gained a significant research interest recently. To enhance the performance of Multi-view Action Recognition, we propose a novel Feature extraction and Feature Selection mechanism that allows building a mutual relationship between the actions sequences of multiple views. The feature extraction considered multiple features which are invariant to scale and orientation. Three different features such as Intensity Features, Orientational Features and Contour Features are used to represent every action. Further, the feature selection is accomplished through self-similarity matrix and is very much helpful in the provision of a perfect discrimination between actions sequences of different views. The proposed method is validated over the standard multi-view IXMAS dataset and experimental results confirm that the proposed method outperforms the conventional approaches with respect to Recognition Accuracy.