The scarcity of the available radio spectrum coupled with the growing popularity of bandwidth intensive mobile video applications poses a huge challenge to network operators. The solution of over-provisioning the network is not economical; hence, an appropriate strategy for scheduling and resource allocation among the users in the system is of crucial importance. This work focuses on scheduling multiple video flows on the downlink of a wireless system based on orthogonal frequency division multiple access (OFDMA), such as Long-Term Evolution (LTE) and LTE-A (LTE-Advanced) standards. We propose a joint multi-user scheduling and multi-user rate adaptation strategy providing an appropriate trade-off between efficiency and fairness, while ensuring high quality of experience (QoE) for the end users. We consider Scalable Video Coding (SVC) which facilitates the truncation of bit streams, thus allowing graceful degradation of video quality in the event of wireless channel variations or network congestion. The proposed scheduler utilizes QoE-aware priority marking, where video layers are mapped to priority classes and targets at minimizing delay bound violations for the most important priority classes under congestion. In order to reduce congestion, we propose multi-user rate adaptation at the MAC layer via a novel dynamic filtering policy for QoE-based priority classes. Simulation results show that the proposed approach delivers to the end users a similar QoE as delivered by the stateof-the-art cross-layer approaches, where extensive cross-layer signaling, additional video rate adaptation modules at the core network, and frequent link probing from the wireless access network to the rate adaptation modules are required. The latter approaches are not implemented in real systems due to the aforementioned drawbacks, while our approach can be implemented without major modifications in the standard behavior of existing networks and equipment. The proposed framework can deliver delay-sensitive traffic as well as delay-tolerant best-effort traffic.