Bluetooth Low Energy (BLE) is one of the key technologies empowering the Internet of Things (IoT) for indoor positioning. In this regard, Angle of Arrival (AoA) localization is one of the most reliable techniques because of its low estimation error. BLE-based AoA localization, however, is in its infancy as only recently directionfinding feature is introduced to the BLE specification. Furthermore, AoA-based approaches are prone to noise, multi-path, and path-loss effects. The paper proposes an efficient Convolutional Neural Network (CNN)-based indoor localization framework to tackle these issues specific to BLE-based settings. We consider indoor environments without presence of Line of Sight (LoS) links affected by Additive White Gaussian Noise (AWGN) with different Signal to Noise Ratios (SNRs) and Rayleigh fading channel. Moreover, by assuming a 3-D indoor environment, the destructive effect of the elevation angle of the incident signal is considered on the position estimation. The effectiveness of the proposed CNN-AoA framework is evaluated via an experimental testbed, where In-phase/Quadrature (I/Q) samples, modulated by Gaussian Frequency Shift Keying (GFSK), are collected by four BLE beacons. Simulation results corroborate effectiveness of the proposed CNN-based AoA technique to track mobile agents with high accuracy in the presence of noise and Rayleigh fading channel.