Soft strain sensors based on conductive polymer composites (CPCs) provide a simple and feasible detection tool in wearable electronics, soft machines, electronic skin, etc. However, the CPCs-based soft strain sensors exhibit resistive viscoelasticity (or time-dependent properties) that hinder the intuitive reflection of the accurate strain and a simple calibration process. In this paper, CPCs with different carbon nanotubes (CNTs) and carbon black (CB) contents were prepared, and electro-mechanical experiments were conducted to study the effect of filler dimensionality and content on the resistive viscoelasticity of CPCs, aimed at guiding the fabrication of CPCs with low resistive viscoelasticity. Furthermore, resistive viscoelasticity and mechanical viscoelasticity were compared to study the origin of the resistive viscoelasticity of CPCs. We found that, at the vicinity of their percolation threshold, the CPCs exhibit high resistive viscoelasticity despite their high sensitivity. In addition, the secondary peaks for CB/SR composite were negligible when the CB concentration was low. Generally, compared with one-dimensional CNT-filled CPCs, the zero-dimensional CB-filled CPCs show higher sensitivity, lower resistive hysteresis, lower resistance relaxation ratio, and better cyclic performance, so they are more suitable for sensor usage. By comparing the resistive viscoelasticity and mechanical viscoelasticity of CPCs, it is indicated that, when the concentration of nanoparticles (NPs) approaches the percolation thresholds, the resistive viscoelasticity is mainly derived from the change of conductive network, while when the concentration of NPs is higher, it is primarily due to the unrecoverable deformations inside the material.