Brownian motion has played important roles in many different fields of science since its origin was first explained by Albert Einstein in 1905. Einstein's theory of Brownian motion, however, is only applicable at long time scales. At short time scales, Brownian motion of a suspended particle is not completely random, due to the inertia of the particle and the surrounding fluid. Moreover, the thermal force exerted on a particle suspended in a liquid is not a white noise, but is colored. Recent experimental developments in optical trapping and detection have made this new regime of Brownian motion accessible. This review summarizes related theories and recent experiments on Brownian motion at short time scales, with a focus on the measurement of the instantaneous velocity of a Brownian particle in a gas and the observation of the transition from ballistic to diffusive Brownian motion in a liquid.