Inspired by an analytic construction of Chang, Weinberger and Yu, we define an assembly map in relative geometric K-homology. The properties of the geometric assembly map are studied using a variety of index theoretic tools (for example, the localized index and higher Atiyah-Patodi-Singer index theory). As an application we obtain a vanishing result in the context of manifolds with boundary and positive scalar curvature; this result is also inspired and connected to the work of Chang, Weinberger and Yu. Furthermore, we use results of Wahl to show that rational injectivity of the relative assembly map implies homotopy invariance of the relative higher signatures of a manifold with boundary.