Nonlinearity operation and early gain suppression limit the high‐frequency operation of GaN‐HEMTs. Nonlinear transconductance and resistance drop‐off at relatively large VGS are the major sources for the nonlinear operation of the high electron mobility transistors (HEMTs). In this article, we present the In0.1Ga0.9N channel‐based HEMTs for stable transconductance operation. The device performance is evaluated for both Al0.3Ga0.7N, and In0.17Al0.83N barrier materials with silicon nitride passivation. The In0.17Al0.83N/In0.1Ga0.9N/GaN heterostructure device shows remarkable improvement in gate voltage swing than Al0.3Ga0.7N/In0.1Ga0.9N/GaN. LG 55 nm T‐gate In0.17Al0.83N/In0.1Ga0.9N HEMT exhibited 5 A/mm of maximum drain current density (IDS, max) for 1 V gate bias, 0.72 S/mm of stable transconductance (gm,max), 43.5 V of off‐state breakdown voltage (VBR), and 275/289 GHz of fT/fmax. The HEMT with AlGaN/InGaN/GaN heterostructure showed 2.81 A/mm of maximum drain current density for 1 V gate bias, 0.66 S/mm of stable transconductance, 55.3 V of VBR, and 252/263 GHz of fT/fmax. Moreover, a highest theoretical OIP3 value of 61.2 and 67.6 dBm obtained for AlGaN and InAlN barrier HEMTs, respectively. The proposed InGaN/GaN channel‐based HEMTs are more reliable for high‐frequency linear operation.