An aluminum calorimeter was investigated as a possible real-time dosimeter for electron beams with an ultra-high dose per pulse (DPP), as used in FLASH radiation therapy (a few Gy/pulse). Ionization chambers, the most widely used active dosimeter type in conventional external beam radiation therapy, suffer from large ion recombination losses at these conditions. Passive dosimeters, such as alanine, are independent of dose rate but do not provide real-time read-out. In this work it is shown that the response of alanine is independent of the DPP in the investigated ultra-high DPP range (up to 2.3 Gy/pulse). Alanine dose measurements were then used to determine the ion recombination correction for an Advanced Markus plane-parallel ionization chamber at ultra-high DPP. Ion collection losses larger than 50% were observed. Therefore, ionization chambers are not considered suitable for accurate dosimetry in FLASH radiation therapy. As an alternative, in a second (independent) experiment an aluminum open-to-atmosphere calorimeter, operated in the quasi-adiabatic mode was investigated at ultra-high DPP electron radiation. The beam pulse charge, and thus the DPP, was varied to evaluate the linearity of the calorimeter response in the DPP range between 0.3 and 1.8 Gy/pulse. On average, the standard deviation of the calorimeter response was 0.1%. The response was proportional to the DPP in the investigated range. The average deviation of the linear fit of the calorimeter dose as a function of the beam pulse charge was <0.5%. This preliminary investigation suggests that a simplified calorimeter design is suitable as a dosimeter with real-time read-out for clinical FLASH radiation therapy beams.