The absorbing transfer matrix method (ATMM) is applied to dynamics calculation and optimization design of planar branch piping systems combined with the response surface model or method (RSM) and a new proposed hybrid RSM (HRSM). A main path is chosen to absorb the influences of sub-branches, and the branch point transfer matrix in the main transfer path is established and proved by the ANSYS simulation and experimental results. It shows that the errors between the ATMM and the finite element method are less than 0.36% in the calculation of the natural frequencies of the two-branch piping system, and the ATMM also agrees well with the dynamics response experimental results of the cross-branch piping system. Then, based on the initial design model of a double-branch piping system, first order natural frequency analysis samples are selected for the branch point positions optimization using the uniform design method (UD) and calculated by the ATMM; the minimum value of the RSM, obtained by fitting the samples with a second-order polynomial (SOP) function, is solved as the best design scheme using the genetic algorithm (GA). Finally, for the problem of supporting positions optimization of a T-branch piping system, a HRSM using the SOP function to fit the variation law with different supporting positions in the samples of the dynamics response optimization target, which is the total x-directional vibration acceleration level (TAC) of three supporting points in the frequency 1–300 Hz and the Fourier basis function to fit the relationship of SOP function error with independent variables is proposed combining the Minimize Prediction (MP) adding point criterion to gradually update the samples, until the HRSM reaches stability and obtains the wanted design solution.