Strategies for waste valorisation from domestic and agro-industrial activities must be pursued, and its use as a soil amendment is an interesting possibility. In this four-year study, the effect of applying municipal solid waste (MSW), farmyard manure (FYM), bottom wood ash supplemented with nitrogen (Ash + N), the inorganic fertilization common in the region (50 kg ha−1 N, P2O5 and K2O) (Control) and this inorganic fertilization supplemented with 70 kg N ha−1 (High N) was assessed in a rainfed olive grove planted in a shallow soil with low organic matter and managed with conventional tillage. The High N treatment significantly increased olive yield in comparison to the other treatments (165% more than MSW), and soil available N proved to be the main driver for tree productivity. MSW and FYM increased soil organic matter, as well as the levels of phosphorus and cation exchange capacity, leaving good indications for future production cycles, although during the four years of the study these treatments provided little N to the trees. The High N treatment significantly reduced soil organic matter (63% less than MSW). The result was attributed in part to the soil management system that did not allow the development of herbaceous vegetation, but also to an effect known as “added N interaction”, in which the excess of inorganic N in the soil might have contributed to accelerate the mineralization of native soil organic matter, an aspect that compromises the sustainability of this fertilization strategy. Although MSW and wood ash are sometimes associated with risks of environmental contamination with heavy metals, in this study the levels of heavy metals in soils and in plant tissues were not of concern.