Currently, the significance of social media in disseminating noteworthy information on topics such as health, politics, and the economy is indisputable. During the COVID-19 pandemic, anti-vaxxers use social media to distribute fake news and anxiety-provoking information about the vaccine, which may harm the public. Here, we characterize the psycho-linguistic features of anti-vaxxers on the online social network Twitter. For this, we collected COVID-19 related tweets from February 2020 to June 2021 to analyse vaccination stance, linguistic features, and social network characteristics. Our results demonstrated that, compared to pro-vaxxers, anti-vaxxers tend to have more negative emotions, narrative thinking, and worse moral tendencies. This study can advance our understanding of the online anti-vaccination movement, and become critical for social media management and policy action during and after the pandemic.