During their life span, concrete structures interact with many kinds of external mechanical loads. Most of these loads are considered in advance and result in reversible deformations. Nevertheless, some of the loads cause irreversible, sometimes unnoticed changes below the macroscopic scale depending on the type and dimension of the impact. As the functionality of concrete structures is often relevant to safety and society, their condition must be known and, therefore, assessed on a regular basis. Out of the spectrum of non-destructive monitoring methods, Coda Wave Interferometry using embedded ultrasonic sensors is one particularly sensitive technique to evaluate changes to heterogeneous media. However, there are various influences on Coda waves in concrete, and the interpretation of their superimposed effect is ambiguous. In this study, we quantify the relations of uniaxial compression and uniaxial tension on Coda waves propagating in normal concrete. We found that both the signal correlation of ultrasonic signals as well as their velocity variation directly reflect the stress change in concrete structures in a laboratory environment. For the linear elastic range up to 30% of the strength, we calculated a velocity variation of −0.97‰/MPa for compression and 0.33%/MPa for tension using linear regression. In addition, these parameters revealed even weak irreversible changes after removal of the load. Furthermore, we show the time-dependent effects of shrinkage and creep on Coda waves by providing the development of the signal parameters over time during half a year together with creep recovery. Our observations showed that time-dependent material changes must be taken into account for any comparison of ultrasonic signals that are far apart in time. The study’s results demonstrate how Coda Wave Interferometry is capable of monitoring stress changes and detecting even small-size microstructural changes. By indicating the stated relations and their separation from further impacts, e.g., temperature and moisture, we anticipate our study to contribute to the qualification of Coda Wave Interferometry for its application as an early-warning system for concrete structures.