Just for the glory of the archive. I've tested this feature in production before landing.

Yep. In all cases it's at least a full-table-scan on file_attachment. Probably a bit more RAM-efficient with the NOT EXISTS thing since it does not need to have all file PHIDs in memory.

Tested again. Double-accept.

We can take inspiration from Phame (?)

I'd be worried about DB performance running such a query as our DB has dozens of millions of files... (This is basically SELECT COUNT(*) FROM file_attachment fa WHERE fa.filePHID NOT IN (SELECT f.phid FROM file); in my understanding?)

Unrelated but @aklapper I'm curious about the situation in WMF. If you can maybe report something like this:

git rebase master

A bit more than Normal, since it reflects on a database with orphan elements that creates "ghosts in the UI".

Relevant old upstream comment (from an unrelated task) that describes this problem as "ghosts in the UI":

Thanks again. My only concern is that, if somebody has a 1 trillion file(s) then the patch may lock the involved tables for an unexpected long amount of time, and you cannot really undo, without causing a massive implicit rollback.

git rebase master

Just as clarification for myself (maybe already obvious to others):

Minor clarification.

See also note in https://we.phorge.it/D25475#inline-3542 - looks like "creating a transformation" is done in many places.

In D25051#15888, @valerio.bozzolan wrote:

I'm a little worried about this migration; Do we have garbage collector for this? it would be safer to let it run.

You are concerned for performance reasons because it doesn't limit the results. Right?

I'm a little worried about this migration; Do we have garbage collector for this? it would be safer to let it run.

Gotcha. Probably this is generating the attachment transactions but should be "ignored on no effects":

This feature is not complete also for Conpherence AFAIK