The human structural brain network, or connectome, has a rich-club organization with a small number of brain regions showing high network connectivity, called hubs. Hubs are centrally located in the network, biological costly, and might be critical for human cognition. Normal aging is accompanied by cognitive decline, including processing speed, and has been linked to energy depletion in the neurons at a molecular level. However, it is still unclear how age affects hub connections in human connectome. Using a large healthy sample (N=137) with wide age range (25-85 years), we found that the anatomical locations of hubs are relatively age-invariant. However, there were wide-spread age effects on the connectivity strength in the connectome. Critically, these age effects were larger in connections within hubs compared to more peripheral brain regions. These findings were supported by both cross-sectional and longitudinal (N=83) analyses. Moreover, results demonstrated associations between connectivity strength and processing speed, which were more concentrated in hub connections than chance level, and the connectivity strength in hub connections mediated the age-effects on processing speed. Overall, our findings indicate that the hub connections are especially affected by aging, and their vulnerability to the age effects might explain the lower processing speed in older adults.