Smart antenna systems play a pivotal role in modern wireless communication by dynamically adjusting antenna radiation patterns to enhance signal reception or transmission. This paper explores the application of adaptive algorithms and machine learning techniques in optimizing the direction of arrival (DOA) estimation for smart antennas. We review related work in the field, including studies on antenna design, gain enhancement, and multiple-input multiple-output (MIMO) systems. The methodology section details the use of algorithms such as Least Mean Squares (LMS) and Long Short-Term Memory (LSTM) networks to improve DOA estimation and beamforming. Our extensive result analysis demonstrates the effectiveness of these algorithms in various scenarios, including different numbers of antennas and angles of signal arrival. Through AOA analysis, we highlight how machine learning and deep learning techniques can significantly enhance the capabilities of smart antenna systems, making them adaptable to diverse signal environments.