Page MenuHomePhorge

Disallow webcrawlers to follow Paste line number anchor links
Closed, ResolvedPublic

Description

Paste provides line anchor links in every single line of a paste, e.g. the line number in line 60 of https://we.phorge.it/P17 links to https://we.phorge.it/P17$60.
It makes no sense for crawlers to follow such links as they'll index the very same document again.
Thus robots.txt should be adjusted accordingly.