rP05f4d5071fdc

Disallow webcrawlers to index Diffusion commits

Summary:
Phorge already sets `Disallow: /diffusion/` and `Disallow: /source/`.
Thus consequently also disallow accessing specific commits via `/r*`.
See https://secure.phabricator.com/T4610 for previous discussions.

Closes T15670

Test Plan:
Go to `/robots.txt` in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.

Reviewers: O1 Blessed Committers, speck

Reviewed By: O1 Blessed Committers, speck

Subscribers: tobiaswiese, valerio.