Disallow webcrawlers to follow Paste line number anchor links
Paste provides line anchor links in every single line of a paste.
If webcrawlers follow these links, they index the very same Paste again.
Thus disallow in robots.txt to reduce unneeded traffic and indexing time.
Go to /robots.txt in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.
Reviewers: O1 Blessed Committers, valerio.bozzolan
Reviewed By: O1 Blessed Committers, valerio.bozzolan
Subscribers: tobiaswiese, valerio.bozzolan, Matthew, Cigaryno
Maniphest Tasks: T15662
Differential Revision: https://we.phorge.it/D25461