The decay of passive scalars is studied in baker's maps with uneven stretching, in the limit of weak diffusion. The map is alternated with diffusion, and three different boundary conditions are employed: zero boundaries, no-flux boundaries, and periodic boundaries. Numerical results are given for scalar decay modes.A set of eigenmode branches and eigenfunctions is also set up for the case of zero diffusion, using a complex variable formulation. The effects of diffusion may then be included by means of a boundary layer theory. Depending on the boundary conditions, the effect of diffusion is to either simply perturb or entirely destroy each zero-diffusion branch.