A semi-parametric mixture of quantile regressions model is proposed to allow regressions of the conditional quantiles, such as the median, on the covariates without any parametric assumption on the error densities. The median as a measure of center is known to be more robust to skewness and outliers than the mean. Modeling the quantiles instead of the mean not only improves the robustness of the model but also reveals a fuller picture of the data by fitting varying quantile functions. The proposed semi-parametric mixture of quantile regressions model is proven to be identifiable under certain weak conditions. A kernel density based EM-type algorithm is developed to estimate the model parameters, while a stochastic version of the EM-type algorithm is constructed for the variance estimation. A couple of simulation studies and several real data applications are conducted to show the effectiveness of the proposed model.