The growth of the tourism industry has greatly boosted the Point-of-Interest (POI) recommendation tasks using Location-based Social Networks (LBSNs). The ever-evolving nature of user preferences poses a major problem. To address this, we propose a Long-term Preference Mining (LTPM) approach that utilizes the temporal recency (TR) measure in the visits along with the location-aware recommendation based on spatial proximity (SP) to the user's location. The temporal dynamics and changing preferences are exploited based on the modified Long Short-term Memory (LSTM) that utilizes the time decay. The spatial considerations are modeled in two aspects: geographical proximity based on enhanced representation learning using orthogonal mapping. Second, the Region-of-Interest (ROI) is based on spatial griding and metric learning to capture the spatial relationships between POIs to enhance the metric space representation. The final recommendations are based on a multi-head attention mechanism that allocates the weights to different features. The combination of three models, called, LTPM-TRSP approach captures the user-POI, POI-POI, and POI-time relationships by focusing on the informative representation of sequential and spatial data. The category-aware final recommendations based on comprehensive historical behavior and geographical context are quite efficacious. The experimentation on three real-world datasets, Gowalla, Foursquare, and Weeplaces, also suggests the potency compared to other state-of-the-art approaches.