As a burgeoning medical imaging method based on hybrid fusion of light and ultrasound, photoacoustic imaging (PAI) has demonstrated high potential in various biomedical applications, especially in revealing the functional and molecular information to improve diagnostic accuracy. However, stemming from weak amplitude and unavoidable random noise, caused by limited laser power and severe attenuation in deep tissue imaging, PA signals are usually of low signal‐to‐noise ratio, and reconstructed PA images are of low quality. Despite that conventional Kalman filter (KF) can remove Gaussian noise in time domain, it lacks adaptability in real‐time estimation due to its fixed model. Moreover, KF‐based denoising algorithm has not been applied in PAI before. In this paper, we propose an adaptive modified KF (MKF) targeted at PAI denoising by tuning system noise matrix Q and measurement noise matrix R in the conventional KF model. Additionally, in order to compensate the signal skewing caused by MKF, we cascade the backward part of Rauch–Tung–Striebel smoother, which utilizes the newly determined Q. Finally, as a supplement, we add a commonly used differential filter to remove in‐band reflection artifacts. Experimental results using phantom and ex vivo colorectal tissue are provided to prove validity of the algorithm.