The quantum Fisher information (QFI) in SU(2) and SU(1,1) interferometers was considered, and the QFI-only calculation was overestimated. In general, the phase estimation as a two-parameterestimation problem, and the quantum Fisher information matrix (QFIM) is necessary. In this paper, we theoretically generalize the model developed by Escher et al [Nature Physics 7, 406 (2011)] to the QFIM case with noise and study the ultimate precision limits of SU(2) and SU(1,1) interferometers with photon losses because photon losses as a very usual noise may happen to the phase measurement process. Using coherent state ⊗ squeezed vacuum state as a specific example, we numerically analyze the variation of the overestimated QFI with the loss coefficient or splitter ratio, and find its disappearance and recovery phenomenon. By adjusting the splitter ratio and loss coefficient the optimal sensitivity is obtain, which is beneficial to quantum precision measurement in a lossy environment.