The success of a dental implant depends on its osseointegration, an important feature of the implant biocompatibility. In this study, two distinct sol-gel hybrid coating formulations [50% methyltrimethoxysilane: 50% 3-glycidoxypropyl-trimethoxysilane (50M50G) and 70% methyltrimethoxysilane with 30% tetraethyl orthosilicate (70M30T)] were applied onto titanium implants. To evaluate their osseointegration, in vitro and in vivo assays were performed. Cell proliferation and differentiation in vitro did not show any differences between the coatings. However, four and eight weeks after in vivo implantation, the fibrous capsule area surrounding 50M50G-implant was 10 and 4 times, respectively, bigger than the area of connective tissue surrounding the 70M30T treated implant. Thus, the in vitro results gave no prediction or explanation for the 50M50G-implant failure in vivo. We hypothesized that the first protein layer adhered to the surface may have direct implication in implant osseointegration, and perhaps correlate with the in vivo outcome. Human serum was used for adsorption analysis on the biomaterials, the first layer of serum proteins adhered to the implant surface was analyzed by proteomic analysis, using mass spectrometry (LC-MS/MS). From the 171 proteins identified; 30 proteins were significantly enriched on the 50M50G implant surface. This group comprised numerous proteins of the immune complement system, including several subcomponents of the C1 complement, complement factor H, C4b-binding protein alpha chain, complement C5 and C-reactive protein. This result suggests that these proteins enriched in 50M50G surface might trigger the cascade leading to the formation of the fibrous capsule observed. The implications of these results could open up future possibilities to predict the biocompatibility problems in vivo. © 2017 Wiley Periodicals, Inc. J Biomed Mater Res Part B: Appl Biomater, 106B: 1477-1485, 2018.