We theoretically study the photoelectron angular distribution (PAD) from the two-photon single ionization of H and He by femtosecond and attosecond extreme-ultraviolet pulses, based on the time-dependent perturbation theory and simulations with the full time-dependent Schrödinger equation. The PAD is formed by the interference of the s and d continuum wave packets, and, thus, contains the information on the relative phase δ and amplitude ratio between them. We find that, when a spectrally broadened femtosecond pulse is resonant with an excited level, the PAD substantially changes with pulse width, since the competition between resonant and nonresonant ionization paths, leading to δ distinct from the scattering phase shift difference, changes with it. In contrast, when the Rydberg manifold is excited, and for the case of above-threshold two-photon ionization, δ and the PAD do not depend much on pulse width, except for the attosecond region. Thus, the Rydberg manifold and the continuum behave similarly in this respect. For a high-harmonic pulse composed of multiple harmonic orders, while the value of δ is different from that for a single-component pulse, the PAD still rapidly varies with pulse width. The present results illustrate a new way to tailor the continuum wave packet.