Calcium carbide residue (CCR) is a solid waste resulting from acetylene gas production. In this study, CCR was used as an alkali activator to prepare fly ash (FA)-based geopolymers without any alkali supplementation. We studied the factors (FA/CCR ratio, curing temperature, and water/binder ratio) influencing the mechanical property of FA/CCR-based geopolymers. The compressive strength results showed that, by optimizing these three factors, the FA/CCR mixture has great potential for use as a cementitious material and geopolymer with a dense microstructure having a maximal compressive strength of 17.5 MPa. The geopolymers’ chemical structure, microstructure, and chemical composition were characterized and determined by a combination of techniques. All these results revealed that amorphous C-(A)-S-H (calcium (aluminate) silicate hydrate) gels mainly formed after geopolymerization resulting from the reaction of FA and CCR. In addition, some crystallines, such as ettringite and monosulfate, were also formed. Further, geopolymers prepared with a suitable FA/CCR ratio (1:1 or 1:2) possessed a compact microstructure because of their sufficient reactive SiO2 and Al2O3 and high-enough alkalinity, responsible for higher content of C-(A)-S-H formation and better mechanical property. Too high curing temperature or water content induced the formation of a loosely bound geopolymer matrix that strongly weakens its mechanical property.