Barriers to the turbulent transport of heat in a convection flow can be identified by Lagrangian trajectories that stay together for a long time and thus probe spatial regions in the bulk of the fluid flow that do not mix effectively with its surroundings. They form Lagrangian coherent sets which we investigate here in direct numerical simulations of three-dimensional Rayleigh-Bénard convection at three different Prandtl numbers. The analysis is based on 524, 288 massless Lagrangian tracer particles which are advected in the time-dependent flow. Clusters of trajectories are identified by the diffusion map approach, which quantifies the connectivity of trajectory segments by a diffusion process on the data, and a subsequent sparse eigenbasis approximation (SEBA) for cluster detection. The diffusion kernel contains a cutoff that is based on the time-averaged distance between mutual Lagrangian tracers in a time window. The numerical simulations are performed in a cell at an aspect ratio Γ = 16, at fixed Rayleigh number Ra = 10 5 , and Prandtl numbers Pr = 0.1, 0.7 and 7. The combination of diffusion map and SEBA leads to a significantly improved cluster identification that is compared with the large-scale patterns in the Eulerian frame of reference. We show that the Lagrangian coherent sets contribute significantly less to the global turbulent heat transfer for all chosen Prandtl numbers as the trajectories in the spatial complement. This is realized by monitoring local Nusselt numbers, a dimensionless measure of heat transfer, along the tracer trajectories.