Page MenuHomePhorge
Create Task
Maniphest T15670

Disallow webcrawlers to index Diffusion repository commits
Open, Needs TriagePublic
Actions

Description

Per old upstream's https://secure.phabricator.com/T4610 ("I suspect no installs are ever interested in spiders generating an index of Diffusion.") and per current upstream's https://we.phorge.it/robots.txt including Disallow: /diffusion/, I propose to also disallow indexing Diffusion URLs like https://we.phorge.it/rP7868ab3754fad13714640451d664d5bc71b7a02f

Revisions and Commits

rP Phorge
D25474rP05f4d5071fdc Disallow webcrawlers to index Diffusion commits

Related Objects

Event Timeline

aklapper created this task.Thu, Nov 16, 09:55
Herald added subscribers: Cigaryno, Matthew, chris, tobiaswiese. · View Herald TranscriptThu, Nov 16, 09:55
aklapper added a revision: D25474: Disallow webcrawlers to index Diffusion commits.Thu, Nov 16, 10:11
valerio.bozzolan added a project: Discussion Needed.Thu, Nov 16, 10:23
valerio.bozzolan subscribed.

This is something that may be not appreciated by some people.

Not everybody likes GitHub and not everybody mirrors things there, so, some public installations may like to have commits indexed on search engines to simplify troubleshooting in general and attract new contributors who are looking for something that is already fixed by a specific commit that has a particular commit message.

Indeed you may say that Phorge already has Differential revisions with that, but any commit does not need to be associated to a Differential revision. So I'm inclined to say that this is not an optimal default.

It should be easy for other installations to do that, but I don't think it's a good default to de-index all commits.

aklapper closed this task as Resolved by committing rP05f4d5071fdc: Disallow webcrawlers to index Diffusion commits.Fri, Nov 17, 07:45
aklapper claimed this task.
aklapper added a commit: rP05f4d5071fdc: Disallow webcrawlers to index Diffusion commits.
aklapper added a comment.Sat, Nov 18, 08:29

Valerio: Uhm, I'm sorry, I had not seen your comment here before I landed the patch (as I had checked my Differential page instead of my notifications).

avivey subscribed.Sun, Nov 19, 09:01

From a product POV, I agree with @valerio.bozzolan - there is (sometimes) some information on commits that would be nice to index in a search engine - comments, mostly.

What's the motive to disallow these from being indexed?

avivey reopened this task as Open.Sun, Nov 19, 09:14