We present new long-baseline spectro-interferometric observations of the Herbig Ae star HD 163296 (MWC 275) obtained in the H and K bands with the AMBER instrument at the VLTI. The observations cover a range of spatial resolutions between ∼3 and ∼12 milliarcseconds, with a spectral resolution of ∼30. With a total of 1481 visibilities and 432 closure phases, they represent the most comprehensive (u, v) coverage achieved so far for a young star. The circumstellar material is resolved at the sub-AU spatial scale and closure phase measurements indicate a small but significant deviation from point-symmetry. We discuss the results assuming that the near-infrared excess in HD 163296 is dominated by the emission of a circumstellar disk. A successful fit to the spectral energy distribution, near-infrared visibilities and closure phases is found with a model in which a dominant contribution to the H and K band emission originates in an optically thin, smooth and point-symmetric region extending from about 0.1 to 0.45 AU. At a distance of 0.45 AU from the star, silicates condense, the disk becomes optically thick and develops a puffed-up rim, whose skewed emission can account for the non-zero closure phases. We discuss the source of the inner disk emission and tentatively exclude dense molecular gas as well as optically thin atomic or ionized gas as its possible origin. We propose instead that the smooth inner emission is produced by very refractory grains in a partially cleared region, extending to at least ∼0.5 AU. If so, we may be observing the disk of HD 163296 just before it reaches the transition disk phase. However, we note that the nature of the refractory grains or, in fact, even the possibility of any grain surviving at the very high temperatures we require (∼2100−2300 K at 0.1 AU from the star) is unclear and should be investigated further.