The challenges of soil degradation and low crop yield are being addressed in East Africa using a number of soil health-promoting farming systems including Conservation Agriculture (CA). CA is based on principles of minimal soil disturbance, continuous soil cover, and crop diversity, though implementation on farms may vary due to local circumstances. This study evaluated the effect of CA, as practiced by farmers, and compared farmer descriptions of soil health with scientific measures. CA and non-CA fields were compared in regions of Ethiopia (23 farms), Kenya (23 farms), and Tanzania (13 farms) with most fields repeatedly sampled for 2 to 3 years. CA and non-CA fields were located within meters, on soils with the same texture, growing similar crops. CA systems used precision planting, mostly with basins. Soil parameters were assessed using a participatory approach where farmers and field technicians collected data. CA practices improved qualitative soil characteristics as sensed by farmers (e.g., hoe-ability, crusting, smell, water infiltration), and this positive response was consistent across most fields. For qualitative data, the logarithm of the odds ratio estimates method was used to test the probability that CA would provide soil improvement. Fields under the CA treatment had a higher probability of rating better than non-CA fields. Quantitative measurements of pH and microbial respiration validated farmer assessments. CA-managed soils were weakly associated with darker colour, indicating higher soil organic matter. Sandy soils did not appear to improve with CA as dramatically as clay and loam soils, suggesting different CA strategies may be required. We discuss both benefits and drawbacks of this type of participatory, on-farm research. We conclude that farmer participation as citizen scientists will advance soil restoration in East Africa and increase the potential for farmer-to-farmer knowledge exchange of soil-improving practices.