In string or M theories, the spontaneous breaking of 10D or 11D Lorentz symmetry is required to describe our space-time. A direct approach to this issue is provided by the IIB matrix model. We study its 4D version, which corresponds to the zero volume limit of 4D super SU (N ) Yang-Mills theory. Based on the moment of inertia as a criterion, spontaneous symmetry breaking (SSB) seems to occur, so that only one extended direction remains, as first observed by Bialas, Burda et al. However, using Wilson loops as probes of space-time we do not observe any sign of SSB in Monte Carlo simulations where N is as large as 48. This agrees with an earlier observation that the phase of the fermionic integral, which is absent in the 4D model, should play a crucial rôle if SSB of Lorentz symmetry really occurs in the 10D IIB matrix model.