Spatial transcriptomics (ST) is a new technology that measures mRNA expression across thousands of spots on a tissue slice, while preserving information about the spatial location of spots. ST is typically applied to several replicates from adjacent slices of a tissue. However, existing methods to analyze ST data do not take full advantage of the similarity in both gene expression and spatial organization across these replicates. We introduce a new method PASTE (Probabilistic Alignment of ST Experiments) to align and integrate ST data across adjacent tissue slices leveraging both transcriptional similarity and spatial distances between spots. First, we formalize and solve the problem of pairwise alignment of ST data from adjacent tissue slices, or layers, using Fused Gromov-Wasserstein Optimal Transport (FGW-OT), which accounts for variability in the composition and spatial location of the spots on each layer. From these pairwise alignments, we construct a 3D representation of the tissue. Next, we introduce the problem of simultaneous alignment and integration of multiple ST layers into a single layer with a low rank gene expression matrix. We derive an algorithm to solve the problem by alternating between solving FGW-OT instances and solving a Non-negative Matrix Factorization (NMF) of a weighted expression matrix. We show on both simulated and real ST datasets that PASTE accurately aligns spots across adjacent layers and accurately estimates a consensus expression matrix from multiple ST layers. PASTE outperforms integration methods that rely solely on either transcriptional similarity or spatial similarity, demonstrating the advantages of combining both types of information.