In this paper, we study the generalized polynomial chaos (gPC) based stochastic Galerkin method for the linear semiconductor Boltzmann equation under diffusive scaling and with random inputs from an anisotropic collision kernel and the random initial condition. While the numerical scheme and the proof of uniform-in-Knudsen-number regularity of the distribution function in the random space has been introduced in [15], the main goal of this paper is to first obtain a sharper estimate on the regularity of the solution-an exponential decay towards its local equilibrium, which then lead to the uniform spectral convergence of the stochastic Galerkin method for the problem under study.We apply the generalized polynomial chaos approach in the stochastic Galerkin (referred as gPC-SG) framework [6,25,34]. Compared with the classical Monte-Carlo method, the gPC-SG approach enjoys a spectral accuracy in the random space-if the solution is sufficient regular-while the Monte-Carlo method converges with only half-th order accuracy. For recent activities for uncertainty quantificaiton in kinetic theory, we refer to a recent review article [11], which surveyed recent results in the study of kinetic equations with random inputs, [15,3,10,33,14,27,22,16,31,20], including their mathematical properties such as regularity and long-time behavior in the random space, construction of efficient stochastic Galerkin methods and handling of multiple scales by s-AP schemes.Recently, the authors in [28] have provided a general framework using hypocoercivity to study general class of linear and nonlinear kinetic equations with uncertainties from the initial data or collision kernels in both incompressible Navier-Stokes and Euler (acoustic) regimes. For initial data near the global Maxwellian, an exponential convergence of the random solution toward the deterministic global equilibrium, a spectral accuracy and exponential decay of the numerical error of the SG system have been established. We also mention some recent work on uncertainty quantification for hyperbolic equations [8,17,35,36] and highly oscillatory transport equations arisen from non-adiabatic transition in quantum dynamics [4].