In this paper, we study the geometric phase in a system formed by two spatially separated cavities interacting with the environment. Each cavity is filled by a linear optical medium and contains a quantum well. For different initial states, the robustness of the generated geometric phase is analyzed under the effects of the optical susceptibility, the dissipation of the cavities, the exciton-cavity and fiber-cavity couplings. Our results show that the geometric phase is extremely sensitive to the effects of the cavityexciton and the fiber-cavity couplings as well as to the optical susceptibility. This opens new routes to understand the storage and manipulation of quantum data in a quantum network.