In the 40 year history of SETI, radio frequency interference (RFI) has proven to be the dominant background in microwave searches. As the SETI community broadens its electromagnetic scope and searches for optical beacons, it must characterize and identify backgrounds for pulsed optical SETI. We must ask the question: What is the "RFI" for pulsed optical SETI? This paper seeks to answer the question by examining the astrophysical, atmospheric, terrestrial, and instrumental sources of optical pulses of nanosecond timescale. Potential astrophysical/atmospheric sources include airglow and scattered zodiacal light, stellar photon pileup, muon events, and cosmic-ray induceď Cerenkov flashes. Terrestrial sources, including lightning and laser communications, appear negligible. Instrumental backgrounds such as scintillation in detector optics and corona breakdown have been the dominant background in our experiments to date, and present significant design challenges for future optical SETI researchers.