Proper formulation of systems containing plasmonic and photochromic units, such as gold nanoparticles and azobenzene derivatives, yields materials and interfaces with synergic functionalities. Moreover, gold nanoparticles are known to accelerate the Z-E isomerization of azobenzene molecules in the dark. However, very little is known about the light-driven, plasmon-assisted Z-E isomerization of azobenzene compounds. Additionally, most of the azobenzene-gold hybrids are prepared with nanoparticles of small, isotropic shapes and azobenzene ligands covalently linked to the surface of nanostructures. Herein, a formulation of a novel system combining azobenzene derivative, gold nanorods, and cellulose nanofibers is proposed. The system's structural integrity relies on electrostatic interactions among components instead of covalent linkage. Cellulose, a robust scaffold, maintains the material's functionality in water and enables monitoring of the material's plasmonic-photochromic properties upon irradiation and at elevated temperatures without gold nanorods' aggregation. Experimental evidence supported by statistical analysis suggests that the optical properties of plasmonic nanometal enable indirect control over the Z-E isomerization of the photochromic component with near-infrared irradiation by triggering the thermoplasmonic effect. The proposed hybrid material's dual plasmonic-photochromic functionality, versatility, and ease of processing render a convenient starting point for further advanced azobenzene-related research and 3D printing of macroscopic light-responsive structures.