The environment we are in can affect our mood and behavior. One environmental factor is weather, which is linked to sentiment as expressed on social media. However, less is known about how integrating changes in weather, along with time and location contextual cues, can improve sentiment detection and understanding. In this paper, we explore the effects of three contextual features--weather, location, and time--on expressed sentiment in social media. Leveraging a large Snapchat dataset, we provide extensive experimental evidence that including contextual features in addition to textual features significantly improves textual sentiment detection performance by 3% over transformer-based language models. Our results also generalize cross-domain to Twitter. Ablation studies indicate the relative importance of weather compared to location and time. We also conduct correlation analyses on 8 million Snapchat posts to highlight the link between past weather and current sentiment, showing that weather has a lasting impact on mood. Users generally exhibit more positive sentiment in better weather conditions as well as in improved weather conditions. Additionally, we show that temperature's link with mood holds after controlling for time or population density, but there exist geographical differences in how temperature affects mood. Our work demonstrates the effectiveness of including external contexts in linguistic tasks and carries design implications for researchers and designers of social media.