The regional mechanical properties of brain tissue are not only key in the context of brain injury and its vulnerability towards mechanical loads, but also affect the behavior and functionality of brain cells. Due to the extremely soft nature of brain tissue, its mechanical characterization is challenging. The response to loading depends on length and time scales and is characterized by nonlinearity, compression-tension asymmetry, conditioning, and stress relaxation. In addition, the regional heterogeneity–both in mechanics and microstructure–complicates the comprehensive understanding of local tissue properties and its relation to the underlying microstructure. Here, we combine large-strain biomechanical tests with enzyme-linked immunosorbent assays (ELISA) and develop an extended type of constitutive artificial neural networks (CANNs) that can account for viscoelastic effects. We show that our viscoelastic constitutive artificial neural network is able to describe the tissue response in different brain regions and quantify the relevance of different cellular and extracellular components for time-independent (nonlinearity, compression-tension-asymmetry) and time-dependent (hysteresis, conditioning, stress relaxation) tissue mechanics, respectively. Our results suggest that the content of the extracellular matrix protein fibronectin is highly relevant for both the quasi-elastic behavior and viscoelastic effects of brain tissue. While the quasi-elastic response seems to be largely controlled by extracellular matrix proteins from the basement membrane, cellular components have a higher relevance for the viscoelastic response. Our findings advance our understanding of microstructure - mechanics relations in human brain tissue and are valuable to further advance predictive material models for finite element simulations or to design biomaterials for tissue engineering and 3D printing applications.