The functional profiles of regions in the ventral occipital-temporal cortex (VTC), a critical region for object visual recognition, are associated with the VTC connectivity patterns to nonvisual regions relevant to the corresponding object domain. However, whether and how whole-brain connections affect recognition behavior remains untested. We directly examined the necessity of VTC connectivity in object recognition behavior by testing 82 patients whose lesion spared relevant VTC regions but affected various white matter (WM) tracts and other regions. In these patients, we extracted the whole-brain anatomical connections of two VTC domain-selective (large manmade objects and animals) clusters with probabilistic tractography, and examined whether such connectivity pattern can predict recognition performance of the corresponding domains with support vector regression (SVR) analysis. We found that the whole-brain anatomical connectivity of large manmade object-specific cluster successfully predicted patients' large object recognition performance but not animal recognition or control tasks, even after we excluded connections with early visual regions. The contributing connections to large object recognition included tracts between VTC-large object cluster and distributed regions both within and beyond the visual cortex (e.g., putamen, superior, and middle temporal gyrus). These results provide causal evidence that the VTC whole-brain anatomical connectivity is necessary for at least certain domains of object recognition behavior.