Traditional von Neumann computing architecture with separated computation and storage units has already impeded the data processing performance and energy efficiency, calling for emerging neuromorphic electronic and optical devices and systems which can mimic the human brain to shift this paradigm. Material-level innovation has become the key component to this revolution of information technology. Chalcogenide phase-change material (PCM) as a well-acknowledged data-storage medium is a promising candidate to tackle this challenge. In this review, the use of PCMs to implement artificial neurons and synapses from both the electronic and optical respects is discussed, and in particular, the structure-property physics and transition dynamics that enable such brain-inspired and in-memory computing applications are emphasized. Recent advances on the atomic-level amorphous and crystalline structures, transition mechanisms, materials optimization and design, neural and synaptic devices, brain-inspired chips, and computing systems, as well as the future opportunities of PCMs, are summarized and discussed.