Performing Sentiment Analysis Using Natural Language Models for Urban Policymaking: An analysis of Twitter Data in Brussels

Abstract

Objectives Mobility is a core challenge to transition towards sustainability. Cities are, therefore, rethinking their mobility to reduce negative externalities such as (greenhouse) gas emissions or congestion. When trying to implement sustainable urban mobility plans, there is often resistance from citizens. This can indicate a disconnect between the public and policymakers due to a lack of participation, coupled with the fact that current data-collection methods often used (such as travel surveys) are limited in scope. Advances in big data analysis and user-generated content provide opportunities to gain deeper insights into citizens' perceptions of mobility policy changes. This paper explores how sentiment analysis through deep learning can be used in transport planning. Materials and Methods In this research, we analyse the sentiments of citizens towards recent changes in mobility policy in Brussels, Belgium, through Twitter data. We analyse 1998 tweets about changing mobility policy in Brussels between July 18th, 2019 (forming of the last Brussels regional government), and December 31st, 2022 (starting date of the analyses). For our analysis, we employ two pre-trained language models: XLM-T and GPT4. Results Our results show that the sentiment with regard to the new mobility interventions is, as reflected by Twitter posts, not overwhelmingly negative. Furthermore, we find that the performance scores of XLM-T when no domain-specific fine-tuning has occurred (zero-shot evaluation) is fairly low (0.48). Once the model is trained on our domain-specific data, it reaches an accuracy of 0.67. When using GPT4, the model reaches an accuracy of 0.66. Additionally, GPT4 seems better suited at identifying mismatched tweets, i.e. tweets using vocabulary that has a different sentiment than the one the tweets expresses (e.g., sarcasm). This might indicate that large language models might be better suited to obtain implicit sentiments expressed in a text. Conclusions From a machine learning perspective, our experiments highlight the difficulty of recognising contextual sentiment (in this case, a sentiment towards changes in mobility policies), which may differ from the sentiment reflected in the vocabulary used. This is especially important if these two sentiments do not correspond, a problem both models struggled with. Additionally, we show that GPT4 can provide additional information when performing sentiment analysis by prompting it to attribute scores to texts. This paper opens new perspectives on understanding and addressing public sentiment in urban mobility policies. The advancements in language models, and the effective integration of user-generated content, can provide policymakers with a more comprehensive understanding of public sentiment, facilitating the implementation of certain policies. Clinical Relevance None.