Universal mobile networks require enhanced capability and appropriate quality of service (QoS) and experience (QoE). To achieve this, Long Term Evolution (LTE) system operators have intensively deployed femtocells (HeNBs) along with macrocells (eNBs) to offer user equipment (UE) with optimal capacity coverage and best quality of service. To achieve the requirement of QoS in the handover stage among macrocells and femtocells we need a seamless cell selection mechanism. Cell selection requirements are considered a difficult task in femtocell-based networks and effective cell selection procedures are essential to reduce the ping-pong phenomenon and to minimize needless handovers. In this study, we propose a seamless cell selection scheme for macrocell-femtocell LTE systems, based on the Q-learning environment. A novel cell selection mechanism is proposed for high-density femtocell network topologies to evaluate the target base station in the handover stage. We used the LTE-Sim simulator to implement and evaluate the cell selection procedures. The simulation results were encouraging: a decrease in the control signaling rate and packet loss ratio were observed and at the same time the system throughput was increased.