Sediment transport by wind or water near the threshold of grain motion is dominated by rare transport events. This intermittency makes it difficult to calibrate sediment transport laws, or to define an unambiguous threshold for grain entrainment, both of which are crucial for predicting sediment transport rates. Intermittency in sediment transport has been observed in many contexts, but few studies have attempted to explain its origins or its impact on transport rates. Here we present a model that captures this intermittency and show that the noisy statistics of sediment transport contain useful information about the sediment entrainment threshold and the variations in driving fluid stress. Using a combination of laboratory experiments and analytical results from the study of stochastic systems we determine the threshold for grain entrainment in a novel way that is independent of any previous method. Furthermore, our analysis reveals a new property, the "bed sensitivity", which can be used to predict conditions under which transport will be intermittent. Our work suggests strategies for improving measurements and predictions of sediment flux and hints that the sediment transport law may change close to the threshold of motion.