We present Keck-MOSFIRE H and K spectra for a sample of 24 candidate quiescent galaxies at 3 < z < 4, identified from their rest-frame UV J colors and photometric redshifts in the ZFOURGE and 3DHST surveys. With median integration times of one hour in H and five in K, we obtain spectroscopic redshifts for half of the sample, using either Balmer absorption lines or nebular emission lines. We confirm the high accuracy of the photometric redshifts for this spectroscopically-confirmed sample, with a median |z phot − z spec |/(1 + z spec ) of 1.2%. Two galaxies turn out to be dusty Hα emitters at lower redshifts (z < 2.5), and these are the only two detected in the sub-mm with ALMA. High equivalent-width [O iii] emission is observed in two galaxies, contributing up to 30% of the K-band flux and mimicking the UV J colors of an old stellar population. This implies a failure rate of only 20% for the UV J selection at these redshifts. Lastly, Balmer absorption features are identified in four galaxies, among the brightest of the sample, confirming the absence of OB stars. We then modeled the spectra and photometry of all quiescent galaxies with a wide range of star-formation histories. We find specific star-formation rates (sSFR) lower than 0.15 Gyr −1 (a factor of ten below the main sequence) for all but one galaxy, and lower than 0.01 Gyr −1 for half of the sample. These values are consistent with the observed Hβ and [O ii] luminosities, and the ALMA non-detections. The implied formation histories reveal that these galaxies have quenched on average 300 Myr prior to being observed, between z = 3.5 and 5, and that half of their stars were formed by z ∼ 5.5 with a mean SFR ∼ 300 M ⊙ /yr. We finally compared the UV J selection to a selection based instead on the sSFR, as measured from the photometry. We find that galaxies a factor of ten below the main sequence are 40% more numerous than UV J-selected quiescent galaxies, implying that the UV J selection is pure but incomplete. Current models fail at reproducing our observations, and underestimate either the number density of quiescent galaxies by more than an order of magnitude, or the duration of their quiescence by a factor two. Overall, these results confirm the existence of an unexpected population of quiescent galaxies at z > 3, and offer the first insights on their formation histories.