We examine the effect of point source confusion on cluster detection in Sunyaev-Zel'dovich (SZ) surveys. A filter matched to the spatial and spectral characteristics of the SZ signal optimally extracts clusters from the astrophysical backgrounds. We calculate the expected confusion (point source and primary cosmic microwave background [CMB]) noise through this filter and quantify its effect on the detection threshold for both single and multiple frequency surveys. Extrapolating current radio counts, we estimate that confusion from sources below ∼100 µJy limits single-frequency surveys to 1σ detection thresholds of Y ∼ 3 × 10 −6 arcmin 2 at 30 GHz and Y ∼ 10 −5 arcmin 2 at 15 GHz (for unresolved clusters in a 2 arcmin beam); these numbers are highly uncertain, and an extrapolation with flatter counts leads to much lower confusion limits. Bolometer surveys must contend with an important population of infrared point sources. We find that a three-band matched filter with 1 arcmin resolution (in each band) efficiently reduces confusion, but does not eliminate it: residual point source and CMB fluctuations contribute significantly to the total filter noise. In this light, we find that a 3-band filter with a low-frequency channel (e.g, 90+150+220 GHz) extracts clusters more effectively than one with a high frequency channel (e.g, 150+220+300 GHz).