This paper presents a deep reinforcement learning (DRL) approach for controlling a compact fiber drawing system. The compact fiber drawing system is smaller and less expensive than industrial draw towers. It is suitable for prototyping novel variable diameter polymer fibers. A controller for the system was developed using DRL. Especially, we focused on regulating the fiber diameter to track non-steady trajectories, where it needs to deal with stochasticity and nonlinear delayed dynamics of the system. The custom DRL-based controller learned to control the process dynamically within approximately two hours of real-time training. This enabled the regulation of the diameter to various trajectories such as step or spline. While a PI feedback controller showed 5.7 seconds of average delay in a step response, the DRL controller showed only -0.7 seconds of average delay. It did not require prior analytical or numerical models of the system. It was also able to track trajectories that it has never faced in the training process.