Driving quality and vehicles safety of hybrid electric vehicles (HEVs) are two hot-topic issues in automobile technology. Nowadays, research focuses to more intelligent and convenient HEVs fault detection methods. This paper will focus on the fault detection of HEV powertrain system with a data-driven algorithm. Orthonormal subspace analysis (OSA) is a newly proposed data-driven method which adds the ability of fault separation. Nonetheless, the linear OSA algorithm cannot effectively detect powertrain system faults, since these faults present complex nonlinear characteristics. A new kernel OSA (KOSA) method is proposed to transform the nonlinear problem into a linear problem through the mapping of kernel function and the dimensionality reduction technique of OSA. Testing results on a nonlinear model and real samples of XMQ6127AGCHEVN61 HEV show that KOSA address the nonlinear problems and it performs better than OSA and kernel principal component analysis (KPCA)