The failure mechanism of hydraulic pump is complex, and its faulty features are frequently submerged in the nonlinear interference caused by various components. The fault diagnosis of hydraulic pump is a challenge in the field of machinery. The conventional fault diagnosis approaches have several drawbacks. First, the operator should be cognizant of the mechanism of hydraulic pump. Second, the procedure is onerous, and has many parameters to set. Third, the shallow classification is weak for this complex problem, which leads to low accuracy rate. This paper developed a new scheme by using improved convolutional neural network. It can be directly used without human intervention, although the operator knows little knowledge about hydraulic pump. Therefore, it is simple to be employed and easy for widely promotion. Validated by fault diagnosis cases of hydraulic pump, the proposed scheme is not only simple for application, but also is superior to other machine learning algorithms, especially when the pump speed varies.