Recent research has demonstrated potential therapeutic effects of hallucinogens, but little is known regarding enduring effects of hallucinogens on human brain structure. Preclinical findings suggest micro-scale structural neuroplastic changes after hallucinogen administration. The current study sought to investigate the association between hallucinogen use, macroscale brain structure, personality, cognitive ability, and illicit drug use in a naturalistic sample. Data from 53 subjects reporting ever having used hallucinogens and 53 approximately hallucinogen-naïve matched controls were drawn from the Nathan Kline Institute-Rockland Sample database. Participants had completed diffusion tensor imaging, psychological, behavioral, and psychiatric assessments. Groups were compared on measures of personality, cognitive ability, history of illicit drug use, and the density of white matter tracts determined from probabilistic tractography. Hallucinogen users reported greater lifetime use of illicit drugs than controls and scored higher on measures of openness to new experiences and cognitive ability. Hallucinogen users also had greater density of structural connectivity in white matter tracts that are thought to support cognition, emotion, and creativity. These findings are consistent with reports that hallucinogen use may lead to shifts in personality as well as multiple cognitive domains. These novel findings provide clues to potential neural mechanisms underlying therapeutic effects of hallucinogens.