Rough surface contact plasticity at microscale and nanoscale is of crucial importance in many new applications and technologies, such as nano-imprinting and nano-welding. This paper summarizes our recent progress in understanding contact plasticity from a multiscale point of view, and also presents our perspectives. We first discuss a contact model based on fractal roughness and continuum plasticity theory. Interestingly, our simple, elastic-plastic contact model of Weierstrass-Archard type gives rise to many practical scaling relations of contact pressure, contact compliance etc. The usefulness of those predictions is discussed for the experiments of indentation and measurements of thermal and electrical properties. A material length scale can be introduced by a nonlocal plasticity theory, or implicitly by dislocation mechanics model. The recent work on micro-plasticity of surface steps gives a variety of surface yielding and hardening behaviors, depending on interface adhesion, roughness features and slip systems. As a consequence, a rough surface contact at small scale can lead to the formation of a boundary layer with sub-layer dislocation structures, which cannot be predicted by existing strain gradient plasticity theories. The micromechanical analysis of surface plasticity could serve as the connection between microscale bulk dislocation plasticity and nanoscale atomistic simulations.