Page MenuHomePhorge

Disallow webcrawlers to index Diffusion commits

Authored by aklapper on Thu, Nov 16, 10:11.
Referenced Files
Unknown Object (File)
Sun, Dec 3, 10:58
Unknown Object (File)
Sun, Dec 3, 10:18
Unknown Object (File)
Sat, Dec 2, 23:59
Unknown Object (File)
Thu, Nov 30, 11:00
Unknown Object (File)
Thu, Nov 30, 00:02
Unknown Object (File)
Thu, Nov 30, 00:02
Unknown Object (File)
Tue, Nov 28, 08:35
Unknown Object (File)
Mon, Nov 27, 15:16
"Evil Spooky Haunted Tree" token, awarded by valerio.bozzolan.



Phorge already sets Disallow: /diffusion/ and Disallow: /source/.
Thus consequently also disallow accessing specific commits via /r*.
See for previous discussions.

Closes T15670

Test Plan

Go to /robots.txt in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.

Diff Detail

rP Phorge
Lint Not Applicable
Tests Not Applicable

Event Timeline

aklapper retitled this revision from Disallow webcrawlers to follow Paste line number anchor links to Disallow webcrawlers to index Diffusion commits.Thu, Nov 16, 10:12
This revision is now accepted and ready to land.Thu, Nov 16, 23:27

(Note this comment)


Good catch, I missed this existing comment in my review. It might be worth digging into the history here to see if there’s more detail on why preventing them from being indexed is less useful or more difficult.

If someone strongly feels that I should revert, please say so - thanks! :)

I'm not an important stakeholder, but I would like to share that in my installation I would like to have commits indexed as default as it happened as default and as it happens in GitHub and GitLab. So I'm now sincerely understanding how to restore the old behavior without keeping my own fork of Phorge if needed.

I don’t think revert I’d needed but the comment should probably be removed or updated. I’d like to understand why it was deemed hard to do but the solution here doesn’t seem that hard. Maybe it’s more difficult than it appears, or was robots.txt standard later updated in a way that makes this easier, or maybe Phab URLs changed in a way that made this easier but this was never updated, etc.