The objective of this study is to propose and test a theoretical framework which integrates the human sentiment reactions on social media in disasters into infrastructure resilience assessment. Infrastructure resilience assessment is important for reducing adverse consequences of infrastructure failures and promoting human well-being in natural disasters. Integrating societal impacts of infrastructure disruptions can enable a better understanding of infrastructure performance in disasters and human capacities under the stress of disruptions. However, the consideration of societal impacts of infrastructure disruptions is limited in existing studies for infrastructure resilience assessment. That is because an integrative theoretical framework for connecting the societal impacts to infrastructure resilience is missing. This study proposed a theoretical framework to examine the relationship between the societal impacts and infrastructure performance in disasters using social media data. Sentiments of human messages related to infrastructure systems are adopted as an indicator of societal impacts of infrastructure disruptions. A case study for electricity and transportation systems in Houston during the 2017 Hurricane Harvey was conducted to illustrate the application of the proposed framework. We find a relation between human sentiment and infrastructure status and validate it by comparing situational information from relevant tweets with official public sources. The findings enable a better understanding of societal expectations and collective sentiments regarding the infrastructure disruptions and improve the visibility of infrastructure performance in Smart City Digital Twin in the context of disasters.