Nowadays, more and more MMICs (Microwave Monolithic Integrated Circuit), such as limiters and switches, are designed to have balanced and unbalanced test pad structures to solve the challenging size restrictions and integration requirements for MMICs. Hybrid balanced and unbalanced RF (Radio Frequency) probes are adopted for an on-wafer test of the heteromorphy structures. The thru standard based on single balanced or unbalanced structures cannot meet the impedance matching requirements of the hybrid RF probes at the same time, which leads to a dramatic decreasing of the calibration accuracy and cannot satisfy the requirement of MMIC test. Therefore, in this paper, the calibration error estimating of hybrid RF probes based on traditional SOLR (Short Open Load Reciprocal) calibration method is performed, and an on-wafer test approach of MMIC based on hybrid balanced and unbalanced RF probes is proposed which combines the OSL (Open Short Load) second-order de-embedding technique with vector error correction and the matrix transformation technique. The calibration reference plane can be accurately shifted to the probe tip with this method, which greatly improves the test accuracy, and an automatic test system is built for this method based on the object-oriented C# language.