In comparison with the traditional controller PID, the fractional order controller PIλDμ is added with two parameters (μ and λ), that increases the flexibility of the controller. The larger adjustable space makes it more favorable to the control of the nonlinear system like the three-phase grid-connected inverter. However, since the fractional calculus operator is an irrational function on the complex plane, it can’t be implemented directly in simulation or in practical applications. In this paper, the advantages of the fractional order controller PIλDμ will be analyzed, then the fractional calculus operator is fitted by the frequency domain analysis method. Based on the vector method, the fractional order controller PIλDμ of the grid-connected inverter is designed. Simultaneously, the optimal controller parameters are searched with the ITAE and IAE as the performance index. And finally, the results are compared with those of the traditional controller PID. In consideration of the defects of fractional algorithm and single discretization method, a hybrid discretization method is proposed in order to ensure that the discretized controller can keep the same time-domain response and frequency characteristics as the designed controller. The experimental results show that the proposed method has the dynamic and static characteristics better than the traditional controller PID, which proves that the application of fractional order controller in three-phase grid-connected inverter is effective and feasible.