Relic gravitational waves (RGWs) generated in the early Universe form a stochastic GW background, which can be directly probed by measuring the timing residuals of millisecond pulsars. In this paper, we investigate the constraints on the RGWs and on the inflationary parameters by the observations of current and potential future pulsar timing arrays. In particular, we focus on effects of various cosmic phase transitions (e.g. e + e − annihilation, QCD transition and SUSY breaking) and relativistic free-streaming gases (neutrinos and dark fluids) in the general scenario of the early Universe, which have been neglected in the previous works. We find that the phase transitions can significantly damp the RGWs in the sensitive frequency range of pulsar timing arrays, and the upper limits of tensor-to-scalar ratio r increase by a factor ∼ 2 for both current and future observations. However, the effects of free-steaming neutrinos and dark fluids are all too small to be detected. Meanwhile, we find that, if the effective equation of state w in the early Universe is larger than 1/3, i.e. deviating from the standard hot big bang scenario, the detection of RGWs by pulsar timing arrays becomes much more promising.