Visible and infrared photons can be detected with a broadband response via the internal photoeffect. By use of plasmonic nanostructures, i.e., nanoantennas, wavelength selectivity can be introduced to such detectors through geometry-dependent resonances. Also, additional functionality, like electronic responsivity switching and polarization detection, has been realized. However, previous devices consisted of large arrays of nanostructures to achieve detectable photocurrents. Here we show that this concept can be scaled down to a single antenna level, resulting in detector dimensions well below the resonance wavelength of the device. Our design consists of a single electrically connected plasmonic nanoantenna covered with a wide-bandgap semiconductor allowing broadband photodetection in the visible/near-infrared via injection of hot carriers. We demonstrate electrical switching of the color sensitivity as well as polarization detection. Our results hold promise for the realization of ultrasmall photodetectors with advanced functionality.