This analysis aims to generate landslide susceptibility maps (LSMs) using various machine learning methods, namely random forest (RF), alternative decision tree (ADTree) and Fisher’s Linear Discriminant Function (FLDA). The results of the FLDA, RF and ADTree models were compared with regard to their applicability for creating an LSM of the Gallicash river watershed in the northern part of Iran close to the Caspian Sea. A landslide inventory map was created using GPS points obtained in a field analysis, high-resolution satellite images, topographic maps and historical records. A total of 249 landslide sites have been identified to date and were used in this study to model and validate the LSMs of the study region. Of the 249 landslide locations, 70% were used as training data and 30% for the validation of the resulting LSMs. Sixteen factors related to topographical, hydrological, soil type, geological and environmental conditions were used and a multi-collinearity test of the landslide conditioning factors (LCFs) was performed. Using the natural break method (NBM) in a geographic information system (GIS), the LSMs generated by the RF, FLDA, and ADTree models were categorized into five classes, namely very low, low, medium, high and very high landslide susceptibility (LS) zones. The very high susceptibility zones cover 15.37% (ADTree), 16.10% (FLDA) and 11.36% (RF) of the total catchment area. The results of the different models (FLDA, RF, and ADTree) were explained and compared using the area under receiver operating characteristics (AUROC) curve, seed cell area index (SCAI), efficiency and true skill statistic (TSS). The accuracy of models was calculated considering both the training and validation data. The results revealed that the AUROC success rates are 0.89 (ADTree), 0.92 (FLDA) and 0.97 (RF) and predication rates are 0.82 (ADTree), 0.79 (FLDA) and 0.98 (RF), which justifies the approach and indicates a reasonably good landslide prediction. The results of the SCAI, efficiency and TSS methods showed that all models have an excellent modeling capability. In a comparison of the models, the RF model outperforms the boosted regression tree (BRT) and ADTree models. The results of the landslide susceptibility modeling could be useful for land-use planning and decision-makers, for managing and controlling the current and future landslides, as well as for the protection of society and the ecosystem.