In transportation, nature, economy, environment, and many other settings, there are multiple simultaneous phenomena happening that are of interest to model and predict. Over the last few years, the traffic data that we have at our disposal have significantly increased, and we have truly entered the era of big data for transportation. Most existing travel demand prediction methods mainly focus on capturing recurrent mobility trends that relate to habitual/routine behavior, and on exploiting short-term correlations with recent observation patterns. However, valuable information that is often available in the form of unstructured data is neglected when attempting to improve forecasting results. Particularly, under non-recurrent conditions, such as large events, or incidents, we need much better models. In this paper, we explore time-series data and semantic information combinations using machine learning and deep learning techniques in the context of creating a prediction model that is able to capture in real-time future stressful situations of the studied transportation system. We apply the proposed approaches in event areas in New York using publicly available taxi data. We empirically show that the proposed models are able to significantly reduce the error in the forecasts. The importance of semantic information is highlighted in all presented methods and the final mean absolute error of our prediction is decreased by 23.8% for a three months testing period.