With the easy access to mobile networks and the proliferation of video applications, video traffic is occupying a great portion of the network traffic, which poses a new challenge of how to alleviate the heavy backhaul traffic and ensure the high quality of experience for video services. As a promising solution towards addressing this challenge, video caching in edge networks has recently received significant attention, which mostly considers the video popularity and the user preference for the video. However, few studies consider the user behavior and the user preference for different parts of the video that indeed have an essential impact on caching efficiency. Hence, this paper proposes a new caching and resource scheduling scheme for adaptive bitrate videos by incorporating these fine-grained factors. We first model the video service problem as a nonlinear integer programming problem, which can be divided into a cache placement problem and an online resource scheduling problem. Then we design efficient algorithms based on several techniques, including greedy strategy, relaxation and rounding, to solve the two problems. Extensive experimental results based on two real-world datasets show that the proposed solution achieves superior performance compared with several state-of-the-art caching approaches.