Membrane-based processes are attractive for treating oily wastewaters. However, membrane fouling due to the deposition of oil droplets on the membrane surface compromises performance. Here, real-time observation of the deposition of oil droplets by direct confocal microscopy was conducted. Experiments were conducted in dead-end and crossflow modes. Base NF 270 nanofiltration membranes as well as membranes modified by grafting poly(N-isopropylacrylamide) chains from the membrane surface using atom transfer radical polymerization were investigated. By using feed streams containing low and high NaCl concentrations, the grafted polymer chains could be induced to switch conformation from a hydrated to a dehydrated state, as the lower critical solution temperature for the grafted polymer chains moved above and below the room temperature, respectively. For the modified membrane, it was shown that switching conformation of the grafted polymer chains led to the partial release of adsorbed oil. The results also indicate that, unlike particles such as polystyrene beads, adsorption of oil droplets can lead to coalescence of the adsorbed oil droplets on the membrane surface. The results provide further evidence of the importance of membrane properties, feed solution characteristics, and operating mode and conditions on membrane fouling.