The role of laser speckle structure (hot spots) and its ponderomotive self-focusing (PSF), in crossed beam energy transfer (CBET), of smoothed laser beams is investigated in an inhomogeneous expanding plasma. Numerical simulations using the code harmony in two spatial dimensions, demonstrate how self-focusing of laser hot spots in crossed beams can significantly affect the transfer of energy from one beam to the other in addition to the stimulated Brillouin scattering (SBS) process. It is shown that for sufficiently intense laser beams, when the laser hot spots exceed the criterion for self-focusing in a plasma with flow, the angular spread of transmitted light beams increases considerably with the intensity, which arises in particular, in expanding plasma where significant beam deflection is observed. It is shown for the first time that besides SBS, the contribution of speckle structure, PSF, and deflections of the intense hot spots in multiple speckle beams to CBET, therefore matters.