Agricultural intensification and the concomitant landscape homogenization is leading to a worldwide decline in farmland biodiversity. Non-crop habitats in agroecosystems may counteract the loss of arthropods such as spiders and thus contribute to sustainable agriculture. However, the effectiveness of field margins and set-aside wildflower-sown patches in maintaining spider diversity is not well understood. Here, we investigated the effects of three different non-crop habitats, namely field margins, set-aside wildflower-sown patches under power poles (‘power pole islands’), and grassland fallows on spider diversity as compared to wheat fields in an agricultural landscape in western Germany. Using pitfall trapping and suction sampling, we show that species richness and overall conservation value were higher in non-crop habitats than in wheat fields. Interestingly, field margins and power pole islands differed from long-term grassland fallows only in conservation value, which was significantly higher in grassland fallows. Species assemblages differed considerably between grassland fallows, field margins and power pole islands, and wheat fields, documenting the added value of using different conservation strategies. Implications for insect conservation Small-scale non-crop habitats adjacent to wheat fields were surprisingly effective in promoting spider diversity in an agricultural landscape, with field margins and power pole islands being equally effective. To maximize overall diversity in agricultural landscapes, we propose a combination of larger long-term fallows and smaller non-crop habitats such as field margins or set-aside wildflower-sown patches.