Fluid-immersed networks and dense suspensions typically reside near a boundary between soft (or fluid-like) and rigid (or solid-like) mechanical regimes. This boundary can be crossed either by varying the concentration or by deformation. Near the onset or loss of rigidity, dissipation-limiting nonaffine rearrangements dominate the macroscopic viscoelastic response, giving rise to diverging relaxation times and power-law rheology. Here, we derive a simple relationship between nonaffinity and excess viscosity in fluid-immersed amorphous materials. We then demonstrate this relationship and its rheological consequences in simulations of stress relaxation in strained filament networks and dense suspensions.