As a pivotal role in the control, optimization, and monitoring of contemporary industrial processes, soft sensors are frequently employed in the prediction of key quality variables. To achieve accurate prediction of key quality variables in industrial processes, a soft sensor modeling method based on the self-organizing fuzzy neural network with the clustering, merging, and splitting scheme (SOFNN-CMS) is proposed. First, the supervised fuzzy C-means clustering algorithm is proposed to identify the appropriate initial center and width of the fuzzy neural network, obtaining appropriate initial fuzzy rules. Then, a neuron merging and splitting strategy is designed to adjust the structure of the fuzzy neural network, by merging and splitting the hidden neurons according to the distance of clusters, increasing the adaptability of the fuzzy neural network. Besides, to accelerate the convergence of estimation errors, an improved Levenberg Marquardt algorithm is utilized to update neural network parameters in the training phase, realizing the soft sensor modeling of key quality variables. The effectiveness of the proposed SOFNN-CMS neural network is demonstrated on two benchmark problems and an industrial debutanizer column. Finally, the experiments showcase that the proposed SOFNN-CMS neural network can obtain better soft sensor modeling performance with a compact structure.