In this paper, we propose a real-time face tracking system using adaptive face detector and the Kalman filter. Basically, the features used for face detection are five types of simple Haar-like features. To only extract the more significant features from these features, we employ principal component analysis (PCA). The extracted features are used for a learning vector of the support vector machine (SVM), which classifies the faces and non-faces. The face detector locates faces from the face candidates separated from the background by using real-time updated skin color information. We trace the moving faces with the Kalman filter, which uses the static information of the detected faces and the dynamic information of changes between previous and current frames. In this experiment, the proposed system showed an average tracking rate of 97.3% and a frame rate of 23.5 frames per s, which can be adapted into a realtime tracking system.