To investigate the role of atmospheric collisions and cosmic ray albedo neutron decay (CRAND) in the dynamics of energetic electrons in the Earth's inner radiation belt during geomagnetic quiet times, a drift‐collision‐source model that includes azimuthal drift, pitch angle diffusion from elastic collision, energy loss from inelastic collision, and a CRAND source is developed. In the model, the bounce‐averaged pitch angle diffusion coefficients and energy loss rates are calculated based on scattering of electrons with neutrals given by the NRLMSISE‐00 model and with ions and electrons given by International Reference Ionosphere (IRI) 2012 model. The electron source rate from CRAND follows the recently developed drift‐source model in Xiang et al. (2019). For 304‐keV quasi‐trapped electrons at L = 1.25, simulation results with CRAND show good agreement with Detection of Electro‐Magnetic Emissions Transmitted from Earthquake Regions satellite observations, confirming that CRAND is the main source for these quasi‐trapped electrons, in contrast to the previous understanding that these quasi‐trapped electrons were formed by wide‐angle scattering of the trapped populations. For trapped electrons, 153, 304, and 509 keV at L < 1.3, the simulation results with only azimuthal drift and atmospheric collisions show a much quicker decrease than observations, while simulation results including a CRAND source are generally comparable to the observations, suggesting that CRAND is an important source of trapped hundreds of kiloelectron‐volt electrons at L < 1.3 during quiet times and provides a baseline for the electron flux even during active times as well. Furthermore, these results suggest that actual radial diffusion rates in the inner belt are lower than previous estimates in which CRAND contributions were not considered.