Traditional on-line surface roughness prediction models are mainly established by surrogate models, which can achieve well prediction accuracies with a fixed tool-workpiece combination. However, a poor prediction accuracy comes to an established model when the tool or workpiece are changed. Then, multiple experiments are required to obtain sufficient new data to establish a new prediction model, increasing the time and economy costs. This paper proposes a data-driven method using transfer learning for on-line classifying the surface roughness under multiple milling conditions. First, a source tool is selected to perform the milling experiments to construct the source data. A stack sparse autoencoder (SSAE) is pre-trained to online classify the surface roughness, where the inputs are the machining parameters and the features derived from the force signals in time and frequency domains. Then, a new tool is selected to perform the milling experiments under fewer milling conditions to construct the target data. The pre-trained SSAE are fine-tuned by re-training the network using the limited target data. Finally, a surface roughness classifier of the target tool is established to adapt to the new milling conditions. Furthermore, a detailed experimental validation is carried out on three different tools of a vertical machining center, indicating a significant potential in establishing an accurate surface roughness classifier with limited milling experiments.