Recent near-infrared power-spectra and panchromatic Extragalactic Background Light (EBL) measurements provide upper limits on the integrated near-infrared surface brightness (SB > ∼ 31mag arcsec −2 at 2µm) that may come from Population III (Pop III) stars and possible accretion disks around resulting stellar-mass black holes (BHs) in the epoch of First Light, broadly taken from z 7-17. Physical parameters for zero metallicity Pop III stars at z > ∼ 7 can be estimated from MESA stellar evolution models through helium-depletion, and for BH accretion disks from quasar microlensing results and multicolor accretion models. Second-generation non-zero metallicity stars can form at higher multiplicity, so that BH accretion disks may be fed by Roche-lobe overflow from lower-mass companions in their AGB stage. The near-infrared SB constraints can be used to calculate the number of caustic transits behind lensing clusters that the James Webb Space Telescope (JWST) and the next generation 25-39 m ground-based telescopes may detect for both Pop III stars and stellar mass BH accretion disks. Because Pop III stars and stellar mass BH accretion disks have sizes of a few×10 −11 arcsec at z > ∼ 7, typical caustic magnifications can be µ 10 4 -10 5 , with rise times of hours and decline times of < ∼ 1 year for cluster transverse velocities of v T < ∼ 1000 km s −1 . Microlensing by intracluster medium objects can modify transit magnifications, and lengthen visibility times. Depending on BH masses, accretion-disk radii and feeding efficiencies, stellar-mass BH accretion-disk caustic transits could outnumber those from Pop III stars. To observe Pop III caustic transits directly may require monitoring 3-30 lensing clusters to AB < ∼ 29 mag over a decade or more. Such a program must be started with JWST at the start of Cycle 1, and -depending on the role of microlensing in the Intra Cluster Light (ICL) -should be continued for decades with the next generation 25-39 m ground-based telescopes, where both JWST and the ground-based facilities each will play a unique and strongly complementary role.