In situ tensile tests employing digital image correlation were conducted to study the martensitic transformation of oligocrystalline Fe–Mn–Al–Ni shape memory alloys in depth. The influence of different grain orientations, i.e., near-〈001〉 and near-〈101〉, as well as the influence of different grain boundary misorientations are in focus of the present work. The results reveal that the reversibility of the martensite strongly depends on the type of martensitic evolving, i.e., twinned or detwinned. Furthermore, it is shown that grain boundaries lead to stress concentrations and, thus, to formation of unfavored martensite variants. Moreover, some martensite plates seem to penetrate the grain boundaries resulting in a high degree of irreversibility in this area. However, after a stable microstructural configuration is established in direct vicinity of the grain boundary, the transformation begins inside the neighboring grains eventually leading to a sequential transformation of all grains involved.