Adhesion of a model rubbery material, crosslinked poly(dimethylsiloxane) (PDMS), onto a solid surface was studied by sum-frequency generation spectroscopy and X-ray photoelectron spectroscopy. To do so, here, we have focused on the adhesive deposit and insoluble layer. The former and latter were defined as the residual amount on the substrate after the peeling and residual layer after washing with a good solvent, respectively. The peel strength of a PDMS sheet adhered onto a glass plate increased with the contact time. Both adhesive deposit and insoluble layer also exhibited comparable contact time dependence. Once a hyperbranched polymer (HBP), which was segregated to the adhesive interface, was incorporated into PDMS, the peel strength and adhesive deposit decreased, although the thickness of the insoluble layer remained almost unchanged. These results suggest that the formation of loosely adsorbed chains on the solid surface, which possess not only trains but also many loop portions and tail parts, plays an important role in the macroscopic adhesion behavior of the PDMS sheet and the interfacial segregation of HBP can prevent it.