Foreign exchange (Forex) rate forecasting is presently pursued by many researchers as it plays an important role in financial technology and business. The challenge of Forex research lies in its characteristics, fluctuation, non-linearity, and random walk phenomena. Several related studies generate forecasting signals using fundamental data (FD) and technical indicator data (TI) to support Forex. FD is an indicator of country economic conditions, while TI shows the price pattern-based signal. However, these two indicators pose two major limitations on their deployment. First, modeling a sequential neural network causes gradient vanishing and information loss. Second, FD exerts a significant impact on currency price upon its quarterly update and release. The second limitation is known as FD releasing problem. Moreover, Forex forecasting using FD and TI is typically conducted in an equal aggregation manner, resulting in inaccurate prediction due to unequal data changing frequency. In this work, we propose BERTFOREX, a cascading model for Forex time-series forecasting. The proposed technique uses deep learning Bidirectional Encoder Representations from Transformer (BERT) based on FD and TI data characteristics. The technique first applies FD to extract the hidden patterns over the designated period. Then, these extracted hidden patterns of FD are aggregated as additional weights to TI since FD frequency changes slower than that of TI. This yields a combined aggregated pattern of FD and TI. BERT again applies the aggregated pattern to discover underlying patterns within TI and FD over other influencing days. We demonstrate the efficiency of BERTFOREX aggregated representation using a simple neural network in forecasting. The proposed method outperforms other methods in terms of percentage of correct signals, sensitivity, specificity, precision, and negative predictive value.