In order to promote the effect of college physical education reform, this paper combines the generalized Hough transform model to analyze the visual movement of physical education teaching. The idea proposed in this paper uses the position information of the edge image itself and the direction information of the curve segment to directly eliminate the impossible targets, which fundamentally alleviates the problem of invalid sampling and accumulation. Moreover, this paper greatly constrains the parameter space based on the results of each segment of the curve, which greatly reduces the search burden of high-dimensional parameters, and combines the improved algorithm to construct a sports teaching video action analysis system. The experimental research shows that the visual movement analysis system of physical education teaching considering the generalized Hough transform model proposed in this paper can effectively analyze the sports teaching actions and improve the efficiency of physical education.