Electrocardiogram (ECG) is a commonly used tool in biological diagnosis of heart diseases. ECG allows the representation of electrical signals which cause heart muscles to contract and relax. Recently, accurate deep learning methods have been developed to overcome manual diagnosis in terms of time and effort. However, most of current automatic medical diagnosis use long electrocardiogram (ECG) signals to inspect different types of heart arrhythmia. Therefore, ECG signal files tend to require large storage to store and may cause significant overhead when exchanged over a computer network. This raises the need to come up with effective compression methods for ECG signals. In this work, the authors investigate using BERT (Bidirectional Encoder Representations from Transformers) model, which is a bidirectional neural network that was originally designed for natural language. The authors evaluate the model with respect to its compression ratio and information preservation, and measure information preservation in terms of the of the accuracy of a convolutional neural network in classifying the decompressed signal. The results show that the method can achieve up to 83% saving in storage. Also, the classification accuracy of the decompressed signals is around 92.41%. Furthermore, the method enables the user to balance the compression ratio and the required accuracy of the CNN classifiers.