Phorge already sets Disallow: /diffusion/ and Disallow: /source/.
Thus consequently also disallow accessing specific commits via /r*.
See https://secure.phabricator.com/T4610 for previous discussions.
Closes T15670
Details
Details
- Reviewers
speck - Group Reviewers
O1: Blessed Committers - Maniphest Tasks
- T15670: Diffusion repository commits: avoid to be a black hole for webcrawlers
- Commits
- rP05f4d5071fdc: Disallow webcrawlers to index Diffusion commits
Go to /robots.txt in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.
Diff Detail
Diff Detail
- Repository
- rP Phorge
- Branch
- SpiderDiffusionCommits (branched from master)
- Lint
Lint Passed - Unit
Tests Passed - Build Status
Buildable 945 Build 945: arc lint + arc unit
Event Timeline
| src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php | ||
|---|---|---|
| 14–16 | (Note this comment) | |
| src/applications/system/controller/robots/PhabricatorRobotsPlatformController.php | ||
|---|---|---|
| 14–16 | Good catch, I missed this existing comment in my review. It might be worth digging into the history here to see if there’s more detail on why preventing them from being indexed is less useful or more difficult. | |
Comment Actions
I'm not an important stakeholder, but I would like to share that in my installation https://gitpull.it I would like to have commits indexed as default as it happened as default and as it happens in GitHub and GitLab. So I'm now sincerely trying to understand how to restore the old behavior without keeping my own fork of Phorge if needed.
Comment Actions
I don’t think revert I’d needed but the comment should probably be removed or updated. I’d like to understand why it was deemed hard to do but the solution here doesn’t seem that hard. Maybe it’s more difficult than it appears, or was robots.txt standard later updated in a way that makes this easier, or maybe Phab URLs changed in a way that made this easier but this was never updated, etc.