We have developed a broadband time-resolved multi-channel near-infrared spectroscopy system that can monitor the physiological responses of the adult human brain. This system is composed of a supercontinuum laser for the source part and of an intensified charge-coupled device camera coupled with an imaging spectrometer for the detection part. It allows the detection of the spectral, from 600 to 900 nm, and spatial dimensions as well as the arrival time of photon information simultaneously. We describe the setup and its characterization in terms of temporal instrument response function, wavelength sensitivity, and stability. The ability of the system to detect the hemodynamic response is then demonstrated. First, an in vivo experiment on an adult volunteer was performed to monitor the response in the arm during a cuff occlusion. Second, the response in the brain during a cognitive task was monitored on a group of five healthy volunteers. Moreover, looking at the response at different time windows, we could monitor the hemodynamic response in depth, enhancing the detection of the cortical activation. Those first results demonstrate the ability of our system to discriminate between the responses of superficial and deep tissues, addressing an important issue in functional near-infrared spectroscopy.