Single-walled carbon nanotubes (SWNTs) have shown great potential in various applications attributed to their unique structures and outstanding structure-dependent properties. The structure-controlled growth of SWNTs is a crucial issue for their advanced applications and has been a great challenge in this field for two decades. Metal catalyst-mediated SWNT growth is believed to be very efficient. In this review, progresses in diameter and chirality controlled growth of SWNTs with metal catalysts is summarized from several aspects, including growth mechanism and theory, effects of catalysts, and the chemical vapor deposition conditions. The design, preparation, handling and dispersion, and the size evolution of metal catalysts are all discussed. The influences of growth environment including the type, composition, and pressure/concentration of the carbon source as well as the temperature on the selectivity toward the nanotube structure are analyzed. We also discuss some of the challenges and trends in this field.