This study develops a three-dimensional viscoelastic model using the finite element method to understand the postseismic deformation that followed the 2011 Tohoku-Oki earthquake. The question of understanding which elements of the viscoelastic media affect the surface deformation is of particular importance. We first examined the individual effects of two different viscoelastic media, the mantle wedge and the oceanic mantle, which produce almost opposite deformation patterns. The mantle wedge controls eastward motion, uplift of the Pacific coastal and offshore regions, and extension across a broad area. In contrast, the oceanic mantle controls dominantly offshore westward motion, subsidence across a broad area, minor uplift of the surrounding areas, and contraction offshore. These differences are the most important issues for understanding the viscoelastic relaxation caused by subduction earthquakes. We then developed four different models to clarify which elements of the viscoelastic media affect the observed surface deformation. The simplest model, with uniform viscosity for all viscoelastic media, could explain the horizontal deformation but not the vertical deformation. The second model, with different viscosities for the mantle wedge and the oceanic mantle, could explain the onshore observations but could not explain the seafloor observations. The third model, which includes a thin weak layer beneath the subducting slab, could essentially explain the near-field onshore and seafloor observations but could not explain the far-field data. The final depth-dependent model was able to explain the far-field data as well as the near-field data. In these typical models, it is of particular importance to consider the different viscosities between the mantle wedge and the oceanic mantle and to include a thin weak layer beneath the slab, which has a dramatic impact on the seafloor deformation. Far-field data as well as near-field data are also important for constraining the viscoelastic structure; the former is sensitive to viscoelastic relaxation at greater depths. Clearly, viscoelastic relaxation alone cannot explain the observed deformation. A combined viscoelastic and afterslip model is necessary for constructing a complete postseismic deformation model.