This paper aims to improve the channel estimation (CE) in the indoor visible light communication system. The proposal of this paper deals with a system that depends on a comparison between Deep Neural Network (DNN), Yolo v3, and Kalman filter (KF) algorithm, for two optical modulation techniques; asymmetrically clipped optical-orthogonal frequency-division multiplexing (ACO-OFDM) and direct current optical-orthogonal frequency division multiplexing (DCO-OFDM). The CE can be evaluated by the error rates in the received bits, where increased error means a performance decrease of the system and vice versa. Receiving less errors at the receiver indicates improved CE and system performance. Hence, the main aim of our work is to decrease the error rate by using different estimators. Furthermore, we apply automatic hyper-parameter approach and Bayesian optimization, to Yolo v3 model to improve the system performance and reduce the positioning error. The metric parameter of bit error rate (BER) aims to determine the improvement ratio in different systems. The model in this paper is based on training with OFDM samples of signal with labels which are received and are corresponding to the signals of OFDM. At a BER = 10−3 with DCO-OFDM, the DNN outperforms KF with 1.7 dB (8.09%) at the bit energy per noise $$(E_{b} {/}N_{o} )$$
(
E
b
/
N
o
)
axis. Also, for ACO-OFDM at BER = 10−3, the DNN achieves better results than KF by about 1.9 dB (11.8%) at the $$(E_{b} {/}N_{o} ){ }$$
(
E
b
/
N
o
)
axis. For different values of M in QAM, the DNN outperforms KF for ACO-OFDM by average improvement of ~ 1.2 dB (~ 13%).