php-fpm memory leak

Event Timeline

• s.quinn asked this question.Tue, Sep 24, 07:59

• s.quinn created this object in space S1 Public.

Herald added subscribers: Cigaryno, Matthew. · View Herald TranscriptTue, Sep 24, 07:59

free -h ?

Are you importing some huge git repo or something? I haven't seen a lot of OOM with php-fpm on phorge/phabricator. It's generally not been an issue in the past so this is either a new bug or something specific to the environment. Then again, I've almost always ran Phorge with much more than 8GB of ram available.

Appears to be internal monitoring software causing this (not phorge), thanks for the replies.

• s.quinn closed this question as invalid.Mon, Oct 14, 13:44

• s.quinn reopened this question.Thu, Oct 17, 14:56

• s.quinn added an answer.Thu, Oct 17, 15:02

OK, not the monitoring issue.

Looks to be the git repository updates running regularly. php-fpm will go from 400MB to 7GB in a few minutes - and then eventually release it's memory and act normally. There are usually two php-fpm processes using excessive memory.

Possibly there's a rather large git repository with many files triggering this but I don't see any errors in the logs. We have about 100+ repositories but only 40 repos active every day.

I can't see a way to reduce the repository update frequency, appears to be 15 seconds?

Current memory usage when memory issue is active:

              total        used        free      shared  buff/cache   available
Mem:           15Gi        15Gi       135Mi        38Mi       204Mi        53Mi
Swap:         8.0Gi       6.6Gi       1.4Gi

processes

2432682 apache    20   0 8317756   6.1g  17144 D   0.0  39.4   0:22.46 php-fpm
2433124 apache    20   0 4399712   3.5g  17208 D   0.3  22.6   0:09.76 php-fpm
2432876 apache    20   0 8198256   3.3g  17040 D   0.0  21.3   0:23.17 php-fpm
    986 mysql     20   0 5007228 946164      0 S   0.3   5.9 128:10.96 mysqld
2440767 phab-vcs  20   0  467244  34752  16604 D   0.0   0.2   0:00.13 php
2440743 phab-vcs  20   0  467244  34456  16304 D   0.0   0.2   0:00.14 php
2440755 phab-vcs  20   0  465160  33872  16420 D   0.0   0.2   0:00.12 php
2440678 phab-da+  20   0  465108  33848  16448 D   0.0   0.2   0:00.13 php
2440768 phab-vcs  20   0  465108  33472  16292 D   0.0   0.2   0:00.13 php
2440676 phab-da+  20   0  465108  32956  16448 D   0.0   0.2   0:00.13 php
2440750 phab-vcs  20   0  463064  32924  16676 D   0.0   0.2   0:00.11 php
2440764 phab-vcs  20   0  465108  32712  16360 D   0.0   0.2   0:00.13 php
2440751 phab-vcs  20   0  462916  31968  16340 D   0.0   0.2   0:00.12 php
2440744 phab-vcs  20   0  460868  29960  16416 D   0.3   0.2   0:00.11 php
2440659 phab-da+  20   0  471456  28760   8280 S   0.0   0.2   0:00.35 php
2432558 apache    20   0  648272  28548  19212 S   0.0   0.2   0:01.13 php-fpm
2432781 apache    20   0  648380  27816  18900 S   0.0   0.2   0:01.18 php-fpm
2440628 phab-da+  20   0  471456  26064   3788 S   0.0   0.2   0:00.27 php
2440741 phab-vcs  20   0  456772  25220  14804 S   0.0   0.2   0:00.07 php
2440758 phab-vcs  20   0  456772  24968  16600 S   0.3   0.2   0:00.08 php
2440766 phab-vcs  20   0  456772  24872  16504 S   0.0   0.2   0:00.08 php
2440753 phab-vcs  20   0  456772  24868  16504 S   0.0   0.2   0:00.07 php
2440759 phab-vcs  20   0  456772  24780  16416 S   0.0   0.2   0:00.06 php
2440763 phab-vcs  20   0  456772  24744  16376 S   0.0   0.2   0:00.08 php

When the memory is normal it is as follows, example:

Mem:           15Gi       1.3Gi        13Gi        39Mi       406Mi        13Gi
Swap:         8.0Gi       2.1Gi       5.9Gi

    986 mysql     20   0 5007228 868488      0 S   0.7   5.4 128:31.36 mysqld
2432558 apache    20   0  648488  36988  27260 S   0.0   0.2   0:02.15 php-fpm
2432781 apache    20   0  648572  36888  27040 S   0.0   0.2   0:01.80 php-fpm
2441815 apache    20   0  647708  28160  18240 S   0.0   0.2   0:00.13 php-fpm
2443599 apache    20   0  647708  28152  18240 S   0.0   0.2   0:00.08 php-fpm
2441476 apache    20   0  647708  27932  18220 S   0.0   0.2   0:00.17 php-fpm
    549 root      20   0  299028  17464  14932 S   0.0   0.1   1:09.37 systemd-journal
2214372 phab-ap+  20   0  612524  15116   9540 S   0.0   0.1   0:33.03 node
2232840 phab-da+  20   0  469352  13348   3780 S   0.3   0.1   3:00.29 php
   1330 root      10 -10   57484  12260  10680 S   0.0   0.1   0:00.00 iscsid
1472613 apache    20   0 2718556   8148   4300 S   0.0   0.1   3:05.98 httpd
2214389 phab-da+  20   0  469408   8072   4108 S   0.0   0.1   0:32.91 php
1476317 apache    20   0 2587396   7680    516 S   0.0   0.0   3:37.27 httpd
   1353 root      20   0  268432   6808   6156 S   0.0   0.0   1:00.17 httpd
   1332 root      20   0  398000   6344   5812 S   0.0   0.0   1:31.08 rsyslogd
1472614 apache    20   0 2521860   6116   4528 S   0.0   0.0   2:47.10 httpd
2429315 root      20   0  626732   6056   3620 S   0.0   0.0   0:00.24 php-fpm

Repository update is based on how recent the last commit is, but that's only for collecting the list of commits to analyze.
But that should be done in the Daemons, which should at least recover nicely and not break the web side of the application. I don't know if they should be running as php or php-fpm.

bin/phd status should at least show the PID and the name of the daemon, so that would be a pointer. Most of these operations can also be limited or done with a TTY, for debugging.