Motivated by GW170817 we examine constraints that observations put on the angular structure of long gamma-ray burst (GRB) jets. First, the relatively narrow observed distribution of E X /E γ (the isotropic equivalent early X-ray afterglow to prompt γ-ray energy ratio) implies that at any angle that γ-rays are emitted the Lorentz factor must be high. Specifically, the Lorentz factor of γ-ray emitting material cannot drop rapidly with angle, and must be Γ(θ) ∼ > 50 even if there are angles for which the gamma-ray received energy is lower by three orders of magnitude compared to the jet core. Second, jets with an angular structure of the γray emission that over-produce events with a γ-ray luminosity below the peak of the observed luminosity function are ruled-out. This eliminates models in which the γ-ray energy angular distribution isn't sufficiently steep and the Lorentz factor distribution isn't sufficiently shallow. Finally, models with a steep structure (e.g. Gaussian) which are detected away from the jet core generate afterglow light-curves that were never observed. We conclude that even if the jet kinetic energy distribution drops continuously with latitude, efficient γ-ray emission seems to be restricted to material with Γ ∼ > 50 and is most likely confined to a narrow region around the core. While our study is confined to long GRBs, where the observed sample is larger and more complete, there are indications that similar conclusions may be applicable also to short GRBs. We discuss the possible implications to the γ-rays observed in GRB 170817A.