Casimir-type forces, such as those between two neutral conducting plates, or between a sphere, atom or molecule and a plate have been widely studied and are becoming of increasing significance, for example, in nanotechnology. A key challenge is to better understand, from a fundamental microscopic approach, why the Casimir force is in some circumstances attractive and in others repulsive. Here, we study the Casimir-Polder forces experienced by small quantum systems such as atoms or molecules in an optical cavity. In order to make the problem more tractable, we work in a 1+1 dimensional setting, we take into account only the ground state and first excited state of the atom and we model the electromagnetic field as a scalar field with Dirichlet or Neumann boundary conditions. This allows us to determine the conditions for the Casimir force to be attractive or repulsive for individual atoms, namely through the interplay of paramagnetic and diamagnetic vacuum effects. We also study the microscopic-macroscopic transition, finding that as the number of atoms in the cavity is increased, the atoms start to affect the Casimir force exerted on the cavity walls similarly to a dielectric medium.