- Group Reviewers
O1: Blessed Committers
- Maniphest Tasks
- T15670: Disallow webcrawlers to index Diffusion repository commits
- rP05f4d5071fdc: Disallow webcrawlers to index Diffusion commits
Go to /robots.txt in the web browser.
Cross fingers that more webcrawlers abide by RFC 9309.
Good catch, I missed this existing comment in my review. It might be worth digging into the history here to see if there’s more detail on why preventing them from being indexed is less useful or more difficult.
I'm not an important stakeholder, but I would like to share that in my installation https://gitpull.it I would like to have commits indexed as default as it happened as default and as it happens in GitHub and GitLab. So I'm now sincerely understanding how to restore the old behavior without keeping my own fork of Phorge if needed.
I don’t think revert I’d needed but the comment should probably be removed or updated. I’d like to understand why it was deemed hard to do but the solution here doesn’t seem that hard. Maybe it’s more difficult than it appears, or was robots.txt standard later updated in a way that makes this easier, or maybe Phab URLs changed in a way that made this easier but this was never updated, etc.