Much attention has been focused on the design of low barrier nanomagnets (LBM), whose magnetizations vary randomly in time owing to thermal noise, for use in binary stochastic neurons (BSN) which are hardware accelerators for machine learning. The performance of BSNs depend on two important parameters: the correlation time c associated with the random magnetization dynamics in a LBM, and the spin-polarized pinning current Ip which stabilizes the magnetization of a LBM in a chosen direction within a chosen time. Here, we show that common fabrication defects in LBMs make these two parameters unpredictable since they are strongly sensitive to the defects. That makes the design of BSNs with real LBMs very challenging. Unless the LBMs are fabricated with extremely tight control, the BSNs which use them could be unreliable or suffer from poor yield.Index Terms-Low barrier magnets, binary stochastic neurons, correlation time, pinning currents, effect of defects.