The estimation of phase noise of continuous-variable quantum key distribution protocol with a local local oscillator (LLO CVQKD), as a major process in quantifying the secret key rate, is closely relevant to the intensity of the phase reference. However, the transmission of the phase reference through the insecure quantum channel is prone to be exploited by the eavesdropper (Eve) to mount attacks. Here, we introduce a polarization attack scheme against the phase reference. Presently, in a practical LLO CVQKD system, only part of the phase reference pulses are measured to compensate for the polarization drift of the quantum signal pulses in a compensation cycle due to the limited polarization measurement rate, while the other part of the phase reference pulses are not measured. We show that Eve can control the phase noise by manipulating the polarization direction of the unmeasured phase reference to hide her attack on the quantum signal. Simulations show that Eve can obtain partial or total key rates information shared between Alice and Bob as the transmission distance increases. Improving the polarization measurement rate to 100% or monitoring the phase reference intensity in real-time is of great importance to protect the LLO CVQKD from polarization attack.