Aggressive interactions can lead to a social hierarchy and influence the responses of animal behavior and physiology. However, our understanding on the changes of fish behavior and physiology during the process of social hierarchical formation is limited. To explore the responses of fish behavior and physiology to social hierarchy, we examined the differences in the growth performance, aggression, cortisol level, brain serotonergic activity, and brain dopamine activity between the dominant individuals and the subordinate individuals of black rockfish (Sebastes schlegelii) in two time scenarios. In the short-term contest, the cortisol level and the ratio of telencephalic 5-hydroxyindoleacetic acid (5-HIAA)/5-hydroxytryptamine (5-HT) was significantly higher in subordinate individuals than in dominant individuals. In the long-term contest, the ratios of 5-HIAA/5-HT in all brain regions were significantly higher, and the frequency of aggressive acts were significantly lower in subordinate individuals than in dominant individuals. In contrast, no difference was detected in growth performance. Significant positive correlations between the cortisol level and serotonergic activity were observed in the short-term contest, but the serotonergic activity was negatively correlated with the aggressive behavior in the long-term contest. These results suggest that subordinate hierarchy inhibits aggression but does not impact growth in black rockfish. The cortisol-related change in brain monoaminergic activity could be a potential indicator to predict aggressive behavior in black rockfish in captivity with an obvious social hierarchy.