VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights

Abstract

Recognizing vulnerability is crucial for understanding and implementing targeted support to empower individuals in need. This is especially important at the European Court of Human Rights (ECtHR), where the court adapts convention standards to meet actual individual needs and thus to ensure effective human rights protection. However, the concept of vulnerability remains elusive at the ECtHR and no prior NLP research has dealt with it. To enable future work in this area, we present VECHR, a novel expert-annotated multi-label dataset comprised of vulnerability type classification and explanation rationale. We benchmark the performance of state-of-the-art models on VECHR from both the prediction and explainability perspective. Our results demonstrate the challenging nature of the task with lower prediction performance and limited agreement between models and experts. We analyze the robustness of these models in dealing with out-of-domain (OOD) data and observe limited overall performance. Our dataset poses unique challenges offering a significant room for improvement regarding performance, explainability, and robustness.