This study explores the potential of using physical infrastructure as a “social sensor” for identifying marginalized communities. Prior work tends to explore biases in infrastructure as a retrospective “social autopsy”. Instead, our study aims to create an introspective “social biopsy”, using existing infrastructure gaps to inform how future policy and investment can address existing inequities more sharply and proactively. Specifically, this work explores the possibility of using U.S. county-level broadband penetration rates as a social sensor to predict rates of unemployment amidst the COVID-19 pandemic. The result is a 2 × 2 typology of where broadband as a social sensor is sharper (or coarser), as well as prone to error (either false positives or false negatives). We further explore combining broadband with other forms of physical infrastructure (i.e., bridges, buildings, and WiFi-enabled libraries) to create a sensor “array” to further enhance detection. Overall, this work proposes an “infrastructure-as-sensor” approach to better detect social vulnerability during times of crises in hopes of enhancing resilience through providing services more quickly and precisely to those who most need it.