A parallel fully coupled (monolithic) fluid-structure interaction (FSI) algorithm has been applied to the deformation of red blood cells (RBCs) in capillaries, where cell deformability has significant effects on blood rheology. In the present FSI algorithm, fluid domain is discretized using the side-centered unstructured finite volume method based on the Arbitrary Lagrangian-Eulerian (ALE) formulation; meanwhile, solid domain is discretized with the classical Galerkin finite element formulation for the Saint Venant-Kirchhoff material in a Lagrangian frame. In addition, the compatible kinematic boundary condition is enforced at the fluid-solid interface in order to conserve the mass of cytoplasmic fluid within the red cell at machine precision. In order to solve the resulting large-scale algebraic linear systems in a fully coupled manner, a new matrix factorization is introduced similar to that of the projection method, and the parallel algebraic multigrid solver BoomerAMG is used for the scaled discrete Laplacian provided by the HYPRE library, which we access through the PETSc library. Three important physical parameters for the blood flow are simulated and analyzed: (1) the effect of capillary diameter, (2) the effect of red cell membrane thickness, and(3) the effect of red cell spacing (hematocrit). The numerical calculations initially indicate a shape deformation in which biconcave discoid shape changes to a parachute-like shape. Furthermore, the parachute-like cell shape in small capillaries undergoes a cupcake-shaped buckling instability, which has not been observed in the literature. The instability forms thin riblike features, and the red cell deformation is not axisymmetric but three-dimensional. KEYWORDS buckling instability, compatible interface condition, fluid-structure interaction, monolithic approach, red blood cell deformation Int J Numer Meth Biomed Engng. 2019;35:e3171.wileyonlinelibrary.com/journal/cnm