To explore the behavior and mechanisms of heat transfer deterioration (HTD), the flow and thermal performances of supercritical n-decane in upward vertical tubes are investigated with emphasis on the boundary layer. It is realized that the dramatic variation of the thermophysical properties can cause unconventional heat transfer phenomenon and the change of the density is important in particular. The intensity of the HTD gradually increases once the position of the pseudo-critical temperature lays in the buffer layer region. Correspondingly, the turbulence kinetic energy is significantly suppressed. The large temperature gradient in the buffer layer zone means that the temperature commonly used in existing empirical correlations and based on the wall temperature or bulk temperature is not very appropriate and such an approach leads to poor prediction accuracy of the correlations.