We study the Casimir-Polder potential of a multilevel alkali-metal atom near an optical nanofiber. We calculate the mean potential of the atom in a fine-structure level. We perform numerical calculations for the Casimir-Polder potentials of the ground state and a few low-lying excited states of a rubidium atom. We show that, unlike the potential of the ground state, which is negative and attractive, the potential of a low-lying excited state may take positive values, oscillate around the zero value with a decaying amplitude, and become repulsive in some regions of atom-to-surface distances. We observe that, for a nanofiber with a radius of 200 nm, the potential for the state 8S 1/2 of a rubidium atom achieves a positive peak value of about 17 µK at a distance of about 150 nm from the fiber surface, and becomes rather strongly repulsive in the region of distances from 150 to 400 nm. We also calculate the nanofiber-induced shifts of the transition frequencies of the atomic rubidium D2 and D1 lines. We find that the shifts are negative in the region of short distances, become positive, and oscillate around the zero value with a decaying amplitude in the region of large distances.