PurposeRisk management is one of the most influential parts of project management that has a major impact on the success or failure of projects. Due to the increasing use of information technology in all fields and the high failure rate of software development projects, it is essential to predict the risk level of each project effectively before starting. Therefore, the main purpose of this paper is proposing an expert system to infer about the risk of new banking software development project.Design/methodology/approachIn this research, the risk of software developing projects is considered from four dimensions including risk of cost deviation, time deviation, quality deviation and scope deviation, which is examined by rough set theory (RST). The most important variables affecting the cost, time, quality and scope of projects are identified as condition attributes and four initial decision systems are constructed. Grey system theory is used to cluster the condition attributes and after data discretizing, eight rule models for each dimension of risk as a decision attribute are extracted using RST. The most validated model for each decision attribute is selected as an inference engine of the expert system, and finally a simple user interface is designed in order to predict the risk level of any new project by inserting the data of project attributesFindingsIn this paper, a high accuracy expert system is designed based on the combination of the grey clustering method and rough set modeling to predict the risks of each project before starting. Cross-validation of different rule models shows that the best model for determining cost deviation is Manual/Jonson/ORR model, and the most validated models for predicting the risk of time, quality and scope of projects are Entropy/Genetic/ORR, Manual/Genetic/FOR and Entropy/Genetic/ORR models; all of which are more than 90% accurateResearch limitations/implicationsIt is essential to gather data of previous cases to design a validated expert system. Since data documentation in the field of software development projects is not complete enough, grey set theory (GST) and RST are combined to improve the validity of the rule model. The proposed expert system can be used for risk assessment of new banking software projectsOriginality/valueThe risk assessment of software developing projects based on RST is a new approach in the field of risk management. Furthermore, using the grey clustering for combining the condition attributes is a novel solution for improving the accuracy of the rule models.