In this work, we propose a method to stabilize a nucleus in the framework of lattice density-functional theory (LDFT) by imposing a suitable constraint. Using this method, the shape of critical nucleus and height of the nucleation barrier can be determined without using a predefined nucleus as input. As an application of this method, we study the nucleation behavior of vapor-liquid transition in nanosquare pores with infinite length and relate the observed hysteresis loop on an adsorption isotherm to the nucleation mechanism. According to the dependence of hysteresis and the nucleation mechanism on the fluid-wall interaction, w , in this work, we have classified w into three regions ( w>0.9 , 0.1< or =w< or =0.9 , and w<0.1 ), which are denoted as strongly, moderately, and weakly attractive fluid-wall interaction, respectively. The dependence of hysteresis on the fluid-wall interaction is interpreted by the different nucleation mechanisms. Our constrained LDFT calculations also show that the different transition paths may induce different nucleation behaviors. The transition path dependence should be considered if morphological transition of nuclei exists during a nucleation process.