We have carried out full imaging simulation studies to explore the impact of frequency standards in millimeter and submillimeter very long baseline interferometry (VLBI), focusing on the coherence time and sensitivity. In particular, we compare the performance of the H-maser, traditionally used in VLBI, to that of ultra-stable cryocooled sapphire oscillators over a range of observing frequencies, weather conditions, and analysis strategies. Our simulations show that at the highest frequencies, the losses induced by H-maser instabilities are comparable to those from high-quality tropospheric conditions. We find significant benefits in replacing H-masers with cryocooled sapphire oscillator based frequency references in VLBI observations at frequencies above 175 GHz in sites which have the best weather conditions; at 350 GHz we estimate a 20%-40% increase in sensitivity over that obtained when the sites have H-masers, for coherence losses of 20%-10%, respectively. Maximum benefits are to be expected by using co-located Water Vapor Radiometers for atmospheric correction. In this case, we estimate a 60%-120% increase in sensitivity over the H-maser at 350 GHz.