This exception happens once $rule in the loop foreach ($this->getMarkupRules() as $rule) in PhutilRemarkupBlockRule::applyRules($text) becomes ProjectRemarkupRule. That's where it blows up.

Think I may have found what is causing this. We had a few repositories that auto download/sync from github. The user's key who connects to github was missing so the downloads were giving errors. I think this triggers a memory leak for the php process. Now that I've fixed the key issue no errors now appear and phorge looks to be happier, i.e. memory usage looks minimal all the time (i.e. php-fpm processes using ~100MBytes).
I'm going to keep monitoring for a few weeks.
Here's an example from the log:

(The Arguments have a supports feature for these cases - like https://we.phorge.it/source/arcanist/browse/master/src/workflow/ArcanistLintWorkflow.php$67 . Not sure what it actually does, but 🤷🏻‍♂️)

In D25823#22043, @avivey wrote:

What if the user used the svn checkout svn+ssh://phab@phorge.example.com/source/myrepo/trunk myRepoOnlyTrunk form - are we still able to find the right target when arc browse lol.txt? Is it a common use-case?

I will add a warning about --branch and better management

For testing, I deployed the latest revision of this proposed patch in our downstream instance and created a task via incoming plain text mail with attachments - seems to still work as expected so no obvious breakage (yet): https://phabricator.wikimedia.org/T377859

Fix typo

mimemailparser: Check that PHP mailparse extension is installed

The function name including a negative, “nonempty”, throws me off…

The email body says Herald added a comment.
The email subject line header says [Changed Project Column] (not [Commented On]).
So I wouldn't say it is incorrectly attributed.
AFAIK there are no Herald rules which trigger a separate email notification on its own (but I agree that the described behavior also has confused me in the past) as Herald rules are always triggered by some other previous action.

Try to make this a separate branch

Thanks, little cute Phorge kitten in profile picture <3

Hire someone from the Community seems most optimal for many reasons

Nice, it's a good step towards cleaning up this mess!

It would be nice to have some unit tests for this but that seems like it might be quite a bit of work to implement.

@valerio.bozzolan Proposing to decline in favor of T15940: Update copy of external mimemailparser library / D25829: Update mimemailparser from May 2011 version to 8.0.4.
Let's not increase the diff to upstream (13 years difference) even more.

Confirmed on KDE on Debian 12, everything set to Dutch: LC_ALL evaluates to nl_NL.UTF-8. Fails before applying this change, succeeds afterwards.

I can reproduce this (KDE on Debian 12, everything set to Dutch: LC_ALL evaluates to nl_NL.UTF-8). diff returns localized output, but doesn't when using LC_ALL=C.

Has anybody a terminal with non-English environment and that obtains a non-English output with this command?

Thank you for trying. Let's continue the "how to reproduce" discussion in the bug report T15927

If it's the daemons that are having problems, you can probably see in the /daemons/ dashboard which tasks are causing problems, and narrow it down from there.

ARRRGH, I locally did this on master branch instead of a dedicated branch. Sorry, it's been a while!

For context, the ManiphestTransaction table in my installation has >10mio entries so some charts do not render but time out. One approach could be Materialized Views but I don't see how that's feasible given the "Project" tag parameter coming into play (and having thousands of project tags in my installation), so my vague followup idea for my installation is to completely disable the code which handles the legacy data only (as creating the modern data does not rely on querying the ManiphestTransaction table), and thus deliberately ignore ancient legacy data from before 2018 for the sake of performance. That's easier to achieve when the upstream spaghetti code doesn't constantly switch between handling legacy and modern/synthetic data.

For what it's worth I've applied this to my personal instance which tracks master, and I've foisted it on my work instance too---ostensibly tracking stable there, but ;)

Can you add a swap file to allow memory usage above 8gb? This will have poor performance but it may allow repo processing to continue. Can you tell if there is a repo which is still importing? Normal day to day repo tracking shouldn’t require tons of ram so the only situation that I can imagine needing this much ram is importing a large repository.

What is the normal memory usage for the update daemon during the repository update? It seems to be peaking around 8GB per process for our repositories. Maybe I need to add more RAM, i.e. 32 GB to satisfy this, but that's not easy on AWS.

Repository update is based on how recent the last commit is, but that's only for collecting the list of commits to analyze.
But that should be done in the Daemons, which should at least recover nicely and not break the web side of the application. I don't know if they should be running as php or php-fpm.

OK, not the monitoring issue.

@Iniquity - Problem solved by removing that sentence lol - thanks. Now seems super-freshy

Thanks! :)

@BlankEclair Do you want a little help to land this?

@Iniquity - Problem solved by removing that sentence lol - thanks. Now seems super-freshy

Arcanist is misspelled as "Archanist" in the 3rd step of "Updating" section.

Uh! Out of curiosity, what does the notification look like in your web notifications?

Okay, I got it! For future reference if anyone in a similar situation is stumbling across this and wants a fully working example to go off of, my more-than-a-little-cargo-culted code is currently: