The Japanese gravitational wave observatory KAGRA will be operated at cryogenic temperatures to reduce thermal noise. Four main mirrors and their suspension systems, called cryogenic payloads, will be cooled in the cryostat. Vibrations of the cryostat and the cryocooler can contaminate the output of the detector. One of the noise paths is the heat link made from the pure soft metal between the cryogenic payload and cryocoolers to cool the payload. In order to evaluate this noise amplitude, we measured the vibration of the radiation shield at cryogenic temperatures at the cryostat production site in Yokohama, Japan. For this measurement, we developed cryogenic accelerometers. Based on the result of this measurement, we calculated the noise in the KAGRA interferometer. Our results show that with the current design, the seismic noise goal formulated for KAGRA cannot be achieved. Finally, we present a possible design optimization that is meant to reach the nominal sensitivity of the detector.