We investigated the effect of the optical blurring of X-ray source on digital breast tomosynthesis (DBT) image quality using well-designed DBT simulator and table-top experimental systems. To measure the in-plane modulation transfer function (MTF), we used simulated sphere phantom and Teflon sphere phantom and generated their projection data using two acquisition modes (i.e., step-and-shoot mode and continuous mode). After reconstruction, we measured the in-plane MTF using reconstructed sphere phantom images. In addition, we measured the anatomical noise power spectrum (aNPS) and signal detectability. We constructed simulated breast phantoms with a 50% volume glandular fraction (VGF) of breast anatomy using the power law spectrum and inserted spherical objects with 1 mm, 2 mm, and 5 mm diameters as breast masses. Projection data were acquired using two acquisition modes, and in-plane breast images were reconstructed using the Feldkamp-Davis-Kress (FDK) algorithm. For the experimental study, we used BR3D breast phantom with 50% VGF and obtained projection data using a table-top experimental system. To compare the detection performance of the two acquisition modes, we calculated the signal detectability using the channelized Hotelling observer (CHO) with Laguerre-Gauss (LG) channels. Our results show that spatial resolution of in-plane image in continuous mode was degraded due to the optical blurring of X-ray source. This blurring effect was reflected in aNPS, resulting in large β values. From a signal detectability perspective, the signal detectability in step-and-shoot mode is higher than that in continuous mode for small spherical signals but not large spherical signals. Although the step-and-shoot mode has disadvantage in terms of scan time compared to the continuous mode, scanning in step-and-shoot mode is better for detecting small signals, indicating that there is a tradeoff between scan time and image quality.