Common everyday materials such as textiles, foodstuffs, soil or skin can have complex, mutable and varied appearances. Under typical viewing conditions, most observers can visually recognize materials effortlessly, and determine many of their properties without touching them. Visual material perception raises many fascinating questions for vision researchers, neuroscientists and philosophers, yet has received little attention compared to the perception of color or shape. Here we discuss some of the challenges that material perception raises and argue that further philosophical thought should be directed to how we see materials.