Elliptic flow of direct photons in relativistic heavy ion collisions is believed to be dominated by contribution from thermal radiation of quark gluon plasma up to pT ∼ 5 GeV/c, although other sources start outshining the thermal contribution at already smaller values of pT in the direct photon spectrum. The elliptic flow of thermal photons from ideal hydrodynamics considering a smooth initial density distribution under-predicts the PHENIX direct photon data from 200A GeV Au+Au collisions at RHIC by a large margin in the range 1 < pT < 5 GeV/c. However, a significant enhancement of thermal photon production due to fluctuations in the initial QCD matter density distributions is expected. We show that such fluctuations result in substantially larger photon elliptic flow for pT > 2.5 GeV/c compared to a smooth initial-state-averaged density profile. The results from event-by-event hydrodynamics are found to be sensitive to the fluctuation size parameter. However, the effects of initial state fluctuations are insufficient to account for the discrepancy to the PHENIX data for direct photon elliptic flow. Furthermore, the photon v2 is reduced even more when we include the NLO pQCD prompt photon component. We also calculate the spectra and elliptic flow of thermal photons for 2.76A TeV Pb+Pb collisions at LHC and for the 0-40% centrality bin. Thermal photons from event-by-event hydrodynamics along with prompt photons from NLO pQCD calculations explain the ALICE preliminary direct photon data well in the region pT ≥ 2.5 GeV/c. Similar to RHIC, the elliptic flow results at LHC are again found to be much smaller than the ALICE preliminary v2 data.