We consider the omnipresent class of spatially decaying systems, where the sensing and controls is spatially distributed. This class of systems arises in various applications where there is a notion of spatial distance with respect to which couplings between the subsystems can be quantified using a class of coupling weight functions. We exploit spatial decay property of the dynamics of the underlying system in order to introduce system-oriented sparsity measures for spatially distributed systems. We develop a new mathematical framework, based on notions of quasi-Banach algebras of spatially decaying matrices, to relate spatial decay properties of spatially decaying systems to sparsity features of their underlying information structures. Moreover, we show that the inverse-closedness property of matrix algebras plays a central role in exploiting various structural properties of spatially decaying systems. We show that the quadratically optimal state feedback controllers for spatially decaying systems are sparse and spatially localized in the sense that they have near-optimal sparse information structures. Finally, our results are applied to quantify sparsity and spatial localization features of a class of randomly generated power networks.