We present an experiment exploring the role of visual displacement to simulator sickness in a video see-through head-mounted display (HMD). To identify the effect of visual displacement on simulator sickness, we examined the effect of visual displacement conditions (ranging from 50 to 300 mm) on simulator sickness and investigated the adaptation of simulator sickness over three days. The results indicated that the total symptom score of simulator sickness in the 300 mm visual displacement condition was significantly higher than that of simulator sickness in the other visual displacement conditions. In addition, the total symptom score of simulator sickness became significantly lower over 1-3 days in the 200 mm visual displacement condition and 1-2 days in the 300 mm visual displacement condition, which means that adaptation was found over three days. However, only partial adaptation was shown in the visual displacement 300, thereby suggesting that high sensory conflict in visual displacement 300 results in increased time to adapt. These results indicate that simulator sickness in video see-through HMDs are adaptable over time, which supports previous studies.