Physical exercise classification method by Hidden Markov Model(HMM) is considered in this study. The aim of this study is to discuss the availability of HMM based motion modeling in order to compare human skills. In this paper, a preprocessing technique for observed human motion by Self-Organizing Map(SOM) to label a motion characteristic is proposed. Then, HMM construction method by using Genetic Algorithm(GA) with Baum-Welch algorithm, modified crossover and mutation is introduced. Simulation studies are carried out for bat swing motions. It is shown that the proposed approach has an ability to recognize bat swing motions.