Mid-infrared (IR) nonlinear optical (NLO) materials with high performance are vital to expanding the laser wavelengths into the mid-IR region and have important technological applications in many civil and military fields. For the last two decades metal chalcogenides have attracted great attention since many of them possess a large NLO effect, wide transparent range, moderate birefringence, and high resistance to laser damage. However, the discovery of superior mid-IR NLO metal chalcogenides is still a big challenge mainly due to the difficulty of achieving a good balance between the NLO effect and laser damage threshold (LDT). In this review, metal chalcogenides are catalogued according to the different types of microscopic building blocks. These groups include triangle planar units, tetrahedral metal-centered units, polyhedra with second-order John-Teller cations, and polyhedra with stereochemically active lone electron pairs cations, rare-earth cations, and/ or halogen anions. The determinations of these microscopic structures on mid-IR NLO properties in metal chalcogenides are summarized and analyzed combined with available experimental data and first-principle calculations. From the deduced structure−property relationship, the searching directions for new metal chalcogenides that have good mid-IR NLO performances, especially for achieving the balance between large NLO effect and high LDT, are discussed.