Despite the considerable effort, fast and highly sensitive photodetection is not widely available at the low-photon-energy range (~meV) of the electromagnetic spectrum, owing to the challenging light funneling into small active areas with efficient conversion into an electrical signal. Here, we provide an alternative strategy by efficiently integrating and manipulating at the nanoscale the optoelectronic properties of topological Dirac semimetal PtSe2 and its van der Waals heterostructures. Explicitly, we realize strong plasmonic antenna coupling to semimetal states near the skin-depth regime (λ/104), featuring colossal photoresponse by in-plane symmetry breaking. The observed spontaneous and polarization-sensitive photocurrent are correlated to strong coupling with the nonequilibrium states in PtSe2 Dirac semimetal, yielding efficient light absorption in the photon range below 1.24 meV with responsivity exceeding~0.2 A/W and noise-equivalent power (NEP) less than~38 pW/Hz0.5, as well as superb ambient stability. Present results pave the way to efficient engineering of a topological semimetal for high-speed and low-energy photon harvesting in areas such as biomedical imaging, remote sensing or security applications.