The applicability of Fourier's law to heat transfer problems relies on the assumption that heat carriers have mean free paths smaller than important length scales of the temperature profile. This assumption is not generally valid in nanoscale thermal transport problems where spacing between boundaries is small (o1 mm), and temperature gradients vary rapidly in space. Here we study the limits to Fourier theory for analysing three-dimensional heat transfer problems in systems with an interface. We characterize the relationship between the failure of Fourier theory, phonon mean free paths, important length scales of the temperature profile and interfacial-phonon scattering by time-domain thermoreflectance experiments on Si, Si 0.99 Ge 0.01 , boron-doped Si and MgO crystals. The failure of Fourier theory causes anisotropic thermal transport. In situations where Fourier theory fails, a simple radiative boundary condition on the heat diffusion equation cannot adequately describe interfacial thermal transport.