The axial buckling behavior of nanowires is investigated with a new continuum theory, in which the surface effect of nanomaterials is characterized by the surface energy density. Only the surface energy density of bulk materials and the surface relaxation parameter are involved, instead of the surface elastic constants in the classical surface elasticity theory. Two kinds of nanowires with different boundary conditions are discussed. It is demonstrated that the new continuum theory can predict the buckling behavior of nanowires very well. Similar to the prediction of the classical elasticity theory, the critical compressive load of axial buckling of nanowires predicted by the new continuum theory increases with an increasing characteristic length, such as the diameter or height of nanowires. With the same aspect ratio, a nanowire with a rectangular cross section possesses a larger critical buckling load than that with a circular one. However, the surface effect could enhance the critical buckling load not only for a fixed-fixed nanowire but also for a cantilevered one in contrast to the classical elastic model. All the results predicted by the new continuum theory agree well with predictions by the surface elasticity models. The present research not only verifies the validation of the new continuum theory, but also gives a much more convenient characterization of buckling behaviors of nanowires. This should be helpful for the design of nanodevices based on nanomaterials, for example, nanobeams in NEMS or high-precision instruments.