Control of stochastic nonlinear systems turns out to be notoriously difficult when stochastic uncertainties and time‐varying delays occur simultaneously. This article presents a tractable adaptive control scheme for the stochastic nonlinear system with time‐varying delays. To mitigate the effects of stochastic uncertainties, an adaptive embedded cubature Kalman filter is developed to realize the robust estimation of the state. Unlike the conventional cubature Kalman filter with fixed construction, a semi‐definite programming is designed to adjust the weights of cubature points dynamically. Such programming guarantees the positive definiteness of the error covariance matrix, which enhances the reliability of the filtering procedure. Based on more accurate state estimations, the multidimensional Taylor network (MTN) is utilized to evaluate the dynamic performance under time‐varying delays and approximate the optimal policy in the deterministic policy gradient framework. Adaptive tracking control with high computational efficiency is achieved due to the concise topological structure of MTN. The exponential convergence of the state estimation error and the semi‐globally uniform ultimate boundness of the tracking error are verified theoretically. The effectiveness of the proposed method is confirmed by a numerical simulation based on a practical electric industrial system.