With the development of high-throughput technologies, the accumulation of large amounts of multidimensional genomic data provides an excellent opportunity to study the multilevel biological regulatory relationships in cancer. Based on the hypothesis of competitive endogenous ribonucleic acid (RNA) (ceRNA) network, lncRNAs can eliminate the inhibition of microRNAs (miRNAs) on their target genes by binding to intracellular miRNA sites so as to improve the expression level of these target genes. However, previous studies on cancer expression mechanism are mostly based on individual or two-dimensional data, and lack of integration and analysis of various RNA-seq data, making it difficult to verify the complex biological relationships involved. To explore RNA expression patterns and potential molecular mechanisms of cancer, a network-regularized sparse orthogonal-regularized joint non-negative matrix factorization (NSOJNMF) algorithm is proposed, which combines the interaction relations among RNA-seq data in the way of network regularization and effectively prevents multicollinearity through sparse constraints and orthogonal regularization constraints to generate good modular sparse solutions. NSOJNMF algorithm is performed on the datasets of liver cancer and colon cancer, then ceRNA co-modules of them are recognized. The enrichment analysis of these modules shows that >90% of them are closely related to the occurrence and development of cancer. In addition, the ceRNA networks constructed by the ceRNA co-modules not only accurately mine the known correlations of the three RNA molecules but also further discover their potential biological associations, which may contribute to the exploration of the competitive relationships among multiple RNAs and the molecular mechanisms affecting tumor development.