In structural credit risk models, default events and the ensuing losses are both derived from the asset values at maturity. Hence it is of utmost importance to choose a distribution for these asset values which is in accordance with empirical data. At the same time, it is desirable to still preserve some analytical tractability. We achieve both goals by putting forward an ensemble approach for the asset correlations. Consistently with the data, we view them as fluctuating quantities, for which we may choose the average correlation as homogeneous. Thereby we can reduce the number of parameters to two, the average correlation between assets and the strength of the fluctuations around this average value. Yet, the resulting asset value distribution describes the empirical data well. This allows us to derive the distribution of credit portfolio losses. With Monte-Carlo simulations for the Value at Risk and Expected Tail Loss we validate the assumptions of our approach and demonstrate the necessity of taking fluctuating correlations into account.