Page MenuHomePhorge

Disallow webcrawlers to index Diffusion repository commits
Open, Needs TriagePublic

Description

Per old upstream's https://secure.phabricator.com/T4610 ("I suspect no installs are ever interested in spiders generating an index of Diffusion.") and per current upstream's https://we.phorge.it/robots.txt including Disallow: /diffusion/, I propose to also disallow indexing Diffusion URLs like https://we.phorge.it/rP7868ab3754fad13714640451d664d5bc71b7a02f

Event Timeline

valerio.bozzolan subscribed.

This is something that may be not appreciated by some people.

Not everybody likes GitHub and not everybody mirrors things there, so, some public installations may like to have commits indexed on search engines to simplify troubleshooting in general and attract new contributors who are looking for something that is already fixed by a specific commit that has a particular commit message.

Indeed you may say that Phorge already has Differential revisions with that, but any commit does not need to be associated to a Differential revision. So I'm inclined to say that this is not an optimal default.

It should be easy for other installations to do that, but I don't think it's a good default to de-index all commits.

Valerio: Uhm, I'm sorry, I had not seen your comment here before I landed the patch (as I had checked my Differential page instead of my notifications).

From a product POV, I agree with @valerio.bozzolan - there is (sometimes) some information on commits that would be nice to index in a search engine - comments, mostly.

What's the motive to disallow these from being indexed?