In traditional communication system, information of APP (Application) layer, transport layer and MAC (Media Access Control) layer has not been fully interacted, which inevitably leads to inconsistencies among TCP congestion state, clients' requirements, and resource allocation. To solve the problem, we propose a joint optimization framework, which consists of APP layer, transport layer and MAC layer, to improve the video clients' playback experience and system throughput. First, a client requirement aware autonomous packet drop strategy, based on packet importance, channel condition and playback status, is developed to decrease the network load and the probability of rebuffering events. Further, TCP (Transmission Control Protocol) state aware downlink and uplink resource allocation schemes are proposed to achieve smooth video transmission and steady ACK (Acknowledgement) feedback respectively. For downlink scheme, maximum transmission capacity requirement for each client is calculated based on feedback ACK information from transport layer to avoid allocating excessive resource to the client, whose ACK feedback is blocked due to bad uplink channel condition. For uplink scheme, information of RTO (Retransmission Timeout) and TCP congestion window are utilized to indicate ACK scheduling priority. The simulation results show that our algorithm can significantly improve the system throughput and the clients' playback continuity with acceptable video quality.