Let V be a vertex operator algebra satisfying certain reductivity and finiteness conditions such that C V , the category of V -modules, is a modular tensor category. We study open-closed field algebras over V equipped with nondegenerate invariant bilinear forms for both open and closed sectors. We show that they give algebras over a certain C-extension of the so-called Swiss-cheese partial dioperad, and we can obtain Ishibashi states easily in such algebras. The Cardy condition can be formulated as an additional condition on such open-closed field algebras in terms of the action of the modular transformation S : τ → − 1 τ on the space of intertwining operators of V . We then derive a graphical representation of S in the modular tensor category C V . This result enables us to give a categorical formulation of the Cardy condition and the modular invariance condition for 1-point correlation functions on the torus. Then we incorporate these two conditions and the axioms of the open-closed field algebra over V equipped with nondegenerate invariant bilinear forms into a tensor-categorical notion called the Cardy C V |C V ⊗V -algebra. In the end, we give a categorical construction of the Cardy C V |C V ⊗V -algebra in the Cardy case.