Satellite quantum key distribution is a promising technique that overcomes the limited transmission distance in optical-fiber-based systems. The polarization tracking technique is one of the key techniques in the satellite quantum key distribution. With free-space quantum key distribution between an optical ground station and a satellite, the photon polarization state will be changed according to the satellite movement. To enable polarization based quantum key distribution between mobile terminals, we developed a polarization-basis tracking scheme allowing a common frame of reference to be shared. It is possible to orient two platforms along a common axis by detecting the reference optical signal only on the receiver side with no prior information about the transmitter's orientation. We developed a prototype system for free-space quantum key distribution with the polarization-basis tracking scheme. Polarization tracking performance was 0.092• by conducting quantum key distribution experiments over a 1 km free space between two buildings in a Tokyo suburb.