Leveraging the unique physical properties, two-dimensional (2D) materials have circumvented the disadvantages of conventional epitaxial semiconductors and held great promise for potential optoelectronic applications. So far, two main detector architectures including photodiode based on a van der Waals P-N junction or Schottky junction and phototransistor based on individual 2D materials or hybrids have been well developed. However, a trade-off between responsivity and speed always exists in those technologies thus hindering the overall performance improvement. Here, we propose a new device concept by sandwiching the 2D anisotropic semimetal between p-type and n-type semiconductors in the out-of-plane direction, called PSN architecture, realizing the improvement of each parameter including broad spectral coverage, fast speed, high sensitivity, power-free and polarization-sensitive. We stack the p-type 2H-MoTe 2 , Weyl semimetal 1T-MoTe 2 and n-type SnSe 2 layer-by-layer constructing vertical sandwich structure where the top and bottom layers contribute to the internal built-in electric field, the intermediate layer can facilitate the exciton dissociation and act as infrared polarized light sensitizers. As a result, this PSN device exhibits broadband photo-response from 405 to 1,550 nm without external bias supply. At optical communication band (1,310 nm), operating at self-driven mode and room temperature, the responsivity and detectivity can reach up to 64.2 mA•W -1 and 2.2×10 11 Jones, respectively, along with fast speed on the order of millisecond. Moreover, the device simultaneously exhibits exceptional detection capability for infrared polarized light, demonstrating the anisotropic photocurrent ratio of 1.55 at 1,310 nm and 2.02 at 1,550 nm, which is attributed to the strong in-plane optical anisotropy of middle 1T-MoTe 2 layer. This work develops a new photodetector scheme with novel PSN architecture toward broadband, self-power, polarized light sensing and imaging modules.