The application of machine learning (ML) for the automatic classification of building elements is a powerful technique for ensuring information integrity in building information models (BIMs). Previous work has demonstrated the favorable performance of such models on classification tasks using geometric information. This research explores the hypothesis that incorporating contextual information into the ML models can improve classification accuracy. To test this, we created a graph data structure where each building element is represented as a node assigned with basic geometric information. The connections between the graph nodes (edges) represent the immediate neighbors of that node, capturing the contextual information expressed in the BIM model. We devised a process for extracting graphs from BIM files and used it to construct a graph dataset of over 42,000 building elements and used the data to train several types of ML models. We compared the classification results of models that rely only on geometry, to graph neural networks (GNNs) that leverage contextual information. This work demonstrates that graph-based models for building element classification generally outperform classic ML models. Furthermore, dividing the graphs that represent complete buildings into smaller subgraphs further improves classification accuracy. These results underscore the potential of leveraging contextual information via graphs for advancing ML capabilities in the BIM environment.