Vehicular Ad hoc Network (VANET) has recently received considerable attention both from academia and industry. In VANET, both safety related application and user application are expected to be supported through the communication between vehicles and access points (AP), To deal with the problem of short connectivity duration and low throughput of user service in VANET due to the mobility of vehicles and the limited coverage of APs, relay vehicles (RVs) can be applied in VANET. However, in the case that multiple candidate RVs are available, the problem of selecting the optimal RVs has to be considered. In this paper, utility based relay vehicle selection algorithm is proposed which jointly considers multiple metrics from multiple protocol layers, including characteristics of physical layer, the link status, bandwidth and delay characteristics of relays and user service requirement, and utility functions are applied to outrank the candidate relays for both delay sensitive service and throughput sensitive service, respectively. Simulation results demonstrate that comparing to previous algorithms, the proposed algorithm offers better performance in terms of average throughput, the transmission delay and packet loss could also be reduced.