We performed a fluctuation analysis of the Hubble Extremely Deep Field (XDF) at four optical wavelength bands and found large fluctuations that are significantly brighter than those expected for ordinary galaxies. Good cross-correlations with flat spectra are found down to 0.2 arcsec, indicating the existence of a spatial structure even at the 0.2 arcsec scale. The detected auto and cross-correlations provide a lower limit of 24 nW m −2 sr −1 for the absolute sky brightness at 700 ∼ 900 nm, which is consistent with previous observations. We searched for candidate objects to explain the detected large fluctuation using the catalog of the Hubble Ultra Deep Field (UDF), and found that the surface number density of faint compact objects (FCOs) rapidly increases toward the faint end. Radial profiles of FCOs are indistinguishable from the PSF, and the effective radius based on de Vaucouleur's law is estimated to be smaller than 0.02 arcsec. The SEDs of FCOs follow a power law at optical wavelengths, but show greater emission and structure at λ > 1µm. Assuming that the FCOs are the cause of the excess brightness and fluctuations, the faint magnitude limit is 34.9 mag for the F775W band, and the surface number density reaches 2.6 × 10 3 (arcsec) −2 .Recent γ ray observations require that the redshift of FCOs must be less than 0.1, if FCOs 1 are the origin of the excess optical and infrared background. Assuming that FCOs consist of missing baryons, the mass and luminosity of a single FCO range from 10 2 to 10 3 solar units, and mass-to-luminosity ratio is significantly lower than 1.0 solar unit. The maximum effective radius of an FCO is 4.7 pc. These results and the good correlation between near-infrared and X-ray background indicate that FCOs could be powered by the gravitational energy associated with black holes.