A critical issue in urban cellular automata (CA) modeling concerns the identification of transition rules that generate realistic urban land use patterns. Recent studies have demonstrated that linear methods cannot sufficiently delineate the extraordinary complex boundaries between urban and non-urban areas and as most urban CA models simulate transitions across these boundaries, there is an urgent need for good methods to facilitate such delineations. This paper presents a machine learning CA model (termed MachCA) with nonlinear transition rules based on least squares support vector machines (LS-SVM) to simulate such urban growth. By projecting the input dataset into a high dimensional space using the LS-SVM method, an optimal hyper-plane is constructed to separate the complex boundaries between urban and nonurban land, thus enabling the retrieval of nonlinear CA transition rules. In the MachCA model, the transition rules are yes-no decisions on whether a cell changes its state or not, the rules being dynamically updated for each iteration of the model implementation. The application of the MachCA for simulating urban growth in the Shanghai QingpuSongjiang area in China reveals that the spatial configurations of rural-urban patterns can be modeled. A comparison of the MachCA model with a conventional CA model fitted by logarithmic regression (termed LogCA) shows that the MachCA model produces more hits and less misses and false alarms due to its capability for capturing the spatial complexity of urban dynamics. This results in improved simulation accuracies, although with only less than 1 % deviation between the overall errors produced by the MachCA and LogCA models. Nevertheless, the way MachCA model use in retrieving the transition rules provides a new method for simulating the dynamic process of urban growth.