Climate variability and soil degradation threaten livestock and crop production globally, and the Eastern Cape Province of South Africa is not exempt. However, little is documented on indigenous perceptions and knowledge held by the farmers concerning the interactive impact of climate variability and soil quality on livestock and crop productivity. The current study used semi-structured questionnaires and interviews to gather local knowledge about the impact of soil quality indicators (SQI) and climate variability on farm productivity among 162 farmers in the Raymond Mhlaba municipality of Eastern Cape Province. The results showed that there was no significant difference in the association between the knowledge of SQI and farming experience or age of the farmer, except for soil organic matter (p = 0.047) and soil biology (p = 0.018). The prolonged drought, low rainfall, and poor soil quality were the major causes of the decline in production. Changes in grazing management and the addition of chemical fertilizers were mentioned to perpetuate soil erosion gullies, increase soil acidity and compaction, and lower soil moisture. According to the participants, adopting crop and livestock practices such as rotation, manure, and compost application improved soil moisture, fertility, and general agricultural productivity amid the observed climate variability. Therefore, the results provide a solid background for various stakeholders (i.e., decision-makers, policymakers, and scientists) to create agricultural practices tailored to farmers’ understanding that will assist them in mitigating the effects of climate variability and land degradation.