Towards Accurate and Reliable Energy Measurement of NLP Models

Abstract

Accurate and reliable measurement of energy consumption is critical for making wellinformed design choices when choosing and training large scale NLP models. In this work, we show that existing software-based energy measurements are not accurate because they do not take into account hardware differences and how resource utilization affects energy consumption. We conduct energy measurement experiments with four different models for a question answering task. We quantify the error of existing software based energy measurements by using a hardware power meter that provides highly accurate energy measurements. Our key takeaway is the need for a more accurate energy estimation model that takes into account hardware variabilities and the non-linear relationship between resource utilization and energy consumption. We release the code and data at https://github. com/csarron/sustainlp2020-energy.

IrEne: Interpretable Energy Prediction for Transformers

“…This work, and a follow up software framework called experimentimpact-tracker (Henderson et al, 2020) tracks the resource (i.e., CPU, GPU, memory) utilization of an NLP model and predicts energy consumption as a function of resources. However, our previous study shows that this type of resource utilization only modeling can be highly inaccurate (Cao et al, 2020). This is in part due to the complex relationship between resource utilization and energy consumption.…”

“…On the one hand, measuring energy consumption directly through hardware power monitors is not feasible as it requires exclusive access to the hardware and detailed instrumentation. On the other hand, there are software models that predict energy as a function of resource utilization (Strubell et al, 2019;Henderson et al, 2020) but these energy prediction models are inaccurate (Cao et al, 2020). The inaccuracy stems from the prediction models not accounting for the complex interactions between energy consumption and resource utilization.…”

IrEne-viz: Visualizing Energy Consumption of Transformer Models

“…Henderson (2020) use the experiment-impacttracker software framework to report the aggregated energy of benchmark programs, built on Strubell et al (2019). However, Cao et al (2020) show that this type of resource utilization only modeling can be highly inaccurate. Zhou et al (2020) presents an energy efficient benchmark for NLP models.…”

Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools

“…Measuring this impact is a first step for raising awareness and controlling the impact of NLP experiments and operations. Some guidelines were offered in the SustaiNLP workshop 1 to measure impact, and it was suggested that different methods for measuring environmental impact can lead to different conclusions in terms of algorithm efficiency (Cao et al, 2020).…”