Context. Giant galactic H ii regions surrounding central young clusters show compressed molecular shells, which have broken up into clumps, filaments, and elephant trunks interacting with UV light from central OB stars. Tiny, dense clumps of subsolar mass, called globulettes, form in this environment. Aims. We observe and explore the nature and origin of the infrared emission and extinction in these cool, dusty shell features and globulettes in one H ii region, the Rosette nebula, and search for associated newborn stars.Methods. We imaged the northwestern quadrant of the Rosette nebula in the near-infrared (NIR) through wideband JHKs filters and narrowband H 2 1-0 S(1) and Pβ plus continuum filters using the Son of Isaac (SOFI) instrument at the New Technology Telescope (NTT) at European Southern Observatory (ESO). We used the NIR images to study the surface brightness of the globulettes and associated bright rims. We used the NIR JHKs photometry to create a visual extinction map and to search for objects with NIR excess emission. In addition, archival images from Spitzer Infrared Array Camera (IRAC) and Multiband Imaging Photometer for Spitzer (MIPS) 24 μm and Herschel Photoconductor Array Camera and Spectrometer (PACS) observations, covering several bands in the mid-infrared and far-infrared, were used to further analyze the stellar population, to examine the structure of the trunks and other shell structures and to study this Rosette nebula photon-dominated region in more detail. Results. The globulettes and elephant trunks have bright rims in the Ks band, which are unresolved in our images, on the sides facing the central cluster. An analysis of 21 globulettes, where surface brightness in the H 2 1-0 S(1) line at 2.12 μm is detected, shows that approximately a third of the surface brightness observed in the Ks filter is due to this line: the observed average of the H 2 /Ks surface brightness is 0.26 ± 0.02 in the globulettes' cores and 0.30 ± 0.01 in the rims. The estimated H 2 1-0 S(1) surface brightness of the rims is ∼3−8 × 10 −8 Wm −2 sr −1 μm −1 . The ratio of the surface brightnesses support fluorescence instead of shocks as the H 2 excitation mechanism. The globulettes have number densities of n(H 2 ) ∼ 10 −4 cm −3 or higher. We estimated masses of individual globulettes and compared them to the results from previous optical and radio molecular line surveys. We confirm that the larger globulettes contain very dense cores, that the density is also high farther out from the core, and that their mass is subsolar. Two NIR protostellar objects were found in an elephant trunk and one was found in the most massive globulette in our study.