In this paper, we investigate covert communication over millimeter-wave (mmWave) frequencies. In particular, a mmWave transmitter, referred to as Alice, attempts to reliably communicate to a receiver, referred to as Bob, while hiding the existence of communication from a warden, referred to as Willie. In this regard, operating over mmWave bands not only increases the covertness thanks to directional beams, but also increases the transmission data rates given much more available bandwidths and enables ultra-low form factor transceivers due to the lower wavelengths used compared to the conventional radio frequency (RF) counterpart. We first assume that the transmitter Alice employs two independent antenna arrays in which one of the arrays is to form a directive beam for data transmission to Bob. The other antenna array is used by Alice to generate another beam toward Willie as a jamming signal while changing the transmit power independently across the transmission blocks in order to achieve desired covertness. For this dual-beam setup, we characterize Willie's detection error rate with the optimal detector and the closed-form of its expected value from Alice's perspective. We then derive the closed-form expression for the outage probability of the Alice-Bob link, which enables characterizing the optimal covert rate that can be achieved using the proposed setup. We further obtain tractable forms for the ergodic capacity of the Alice-Bob link involving only one-dimensional integrals that can be computed in closed forms for most ranges of the channel parameters. Finally, we highlight how the results can be extended to more practical scenarios, particularly to cases where perfect information about the location of the passive warden is not available. Our results demonstrate the advantages of covert mmWave communication compared to the RF counterpart. The research in this paper is the first analytical attempt in exploring covert communication using mmWave systems.