An algorithm is proposed for mixing matrix estimation in underdetermined blind source separation (UBSS) based on homogeneous polynomials representation, in order to blindly identify a mixing matrix in the case where the source signal is insufficiently sparse. First, the observed signal subspaces (hyperplanes) are identified by polynomial fitting, differentiation, and spectral clustering. Then, the intersection lines between clustering planes are estimated by using normal vectors of each subspace, which are finally proved to be column vectors of the mixing matrix up to scaling and ordering. Based on the algebraicgeometric theory, the proposed algorithm can improve the accuracy of mixing matrix estimation, without being influenced by convergence. Simulation results indicate that the proposed algorithm has higher estimation accuracy than traditional algorithms. Additionally, the single-source and multi-source points of the source signal can be detected simultaneously, which increases the robustness of the algorithm.