Transcranial direct current stimulation (tDCS) has been extensively used to examine whether neural activities can be selectively increased or decreased with manipulations of current polarity. Recently, the field has reevaluated the traditional anodal-increase and cathodal-decrease assumption due to the growing number of mixed findings that report the effects of the opposite directions. Therefore, the directionality of tDCS polarities and how it affects each individual still remain unclear. In this study, we used a visual working memory (VWM) paradigm and systematically manipulated tDCS polarities, types of different independent baseline measures, and task difficulty to investigate how these factors interact to determine the outcome effect of tDCS. We observed that only low-performers, as defined by their no-tDCS corsi block tapping (CBT) performance, persistently showed a decrement in VWM performance after anodal stimulation, whereas no tDCS effect was found when participants were divided by their performance in digit span. In addition, only the optimal level of task difficulty revealed any significant tDCS effect. All these findings were consistent across different blocks, suggesting that the tDCS effect was stable across a short period of time. Lastly, there was a high degree of intra-individual consistency in one’s responsiveness to tDCS, namely that participants who showed positive or negative effect to anodal stimulation are also more likely to show the same direction of effects for cathodal stimulation. Together, these findings imply that tDCS effect is interactive and state dependent: task difficulty and consistent individual differences modulate one’s responsiveness to tDCS, while researchers’ choices of independent behavioral baseline measures can also critically affect how the effect of tDCS is evaluated. These factors together are likely the key contributors to the wide range of “noises” in tDCS effects between individuals, between stimulation protocols, and between different studies in the literature. Future studies using tDCS, and possibly tACS, should take such state-dependent condition in tDCS responsiveness into account.