The High Voltage Direct Current (HVDC) is an emerging technology for transmitting power over long distances with a higher capacity than the traditional AC systems. The integration of the HVDC systems has demanded changes on the Supervisory, Control and Data Acquisition (SCADA) systems. Several power system applications and toolboxes in the SCADA have to be modified to meet the modern power network characteristics. One of the essential toolboxes is the state estimator, which estimates the network AC and DC systems states. On several occasions, the state estimator fails to deal with severely corrupted data, known as bad data. Therefore, an additional data treatment is required. This paper presents a unified bad data detection block for Weighted Least Squares (WLS) state estimation algorithm suitable for hybrid Voltage Source Converter (VSC)-HVDC/AC transmission systems. The bad data detection block improves the traditional Largest Normalized Residual (LNR) method by integrating the Gaussian Mixture Model (GMM) algorithm. The modifications aim to reduce the time performance of the bad data detection, increase the algorithm robustness, and enhance the state estimation accuracy. The Cigre B4 network is used as a test case to validate this work on a hybrid VSC-HVDC/AC network. UK national grid load profile data is used to construct the simulation measurements set and the GMM model. The work has concluded that the modified GMM-LNR has considerably reduced the bad data detection time and improved the WLS state estimation accuracy.