Accurate modeling of contact stiffness is crucial in predicting the dynamic behavior and chatter vibration of spindletoolholder system for high-speed machining centers. This paper presents a fractal theory-based contact model of spindle-toolholder joint to obtain the contact stiffness and its real contact area. Topography of the contact surfaces of spindle-toolholder joint is fractal featured and determined by fractal parameters. Asperities in micro-scale are considered as elastic or plastic deformation. Then, the contact stiffness, the real contact area, the elastic contact force, and the plastic contact force of the whole contact surface are calculated by integrating the micro asperities. The relationship of the contact stiffness and the drawbar force follows a power law, in which the power index is determined by the fractal parameters. Experiments are conducted to verify the efficiency of the proposed model. The results from the fractal contact model of spindle-toolholder joint have good agreement with those of experiments.