How to set the Pull Time Limit for Mercurial repos?
Closed, ResolvedPublic
Actions

Asked by tinloaf on Apr 26 2023, 08:24.

Details

I have a large Mercurial repository that I want to 'observe'. The repository is too large to be pulled within the default time limit of 15 minutes, therefore the pull times out.

I see that in https://secure.phabricator.com/D19831 , the possibility was added to configure a pull time limit for repositories. However, when I open the management view for that repo, I don't see the 'Limits' panel in the nav bar on the left.

The code responsible for setting this limit (by calling $repository->setCopyTimeLimit()) seems to be in https://we.phorge.it/source/phorge/browse/master/src/applications/diffusion/management/DiffusionRepositoryLimitsManagementPanel.php , which contains this function:

public function shouldEnableForRepository(
  PhabricatorRepository $repository) {
  return $repository->isGit();
}

This probably explains why I don't see this panel for my Mercurial repo. However, setCopyTimeLimit() and get(Effective)CopyTimeLimit() methods are part of the base class for repositories and would also be available for a mercurial repository. In fact, reading through the code, the future that runs the pull is created in the base class, i.e., in the exact same way for Mercurial and git repositories.

Two questions:

1.) Is this function intentionally disabled for Mercurial repos? If not, I could cook up a patch (even though I'm really not a PHP developer) - the question is then whether this should be enabled for *all* repos or whether we just want to add Mercurial repos (which would then only exclude SVN, right?)

2.) The getCopyTimeLimit() function uses $repo->getDetail('limit.copy'); to retrieve a time limit, if one was set. Is it possible as a workaround to manually set that 'detail' on a repository? Would I have to manually modify the database, or does a tool exist that allows this modification?

Event Timeline

Looking at the diff, I think this timeout only applies to Drydock cloning the repository for its builds, not for "observing" repositories - and Drydock might not support Mercurial anyway.

The timeout for the original import is 4 hours, which should be long enough for pretty much anything.

Are you seeing any error/exception log? Is there anything failing?

In the repo management panel, I see this error:

Raw Error
Error updating working copy: Command failed with error #255!
COMMAND
hg --config ui.ssh=/var/www/phorge/phorge/bin/ssh-connect pull -- '********'

STDOUT
pulling from https://<redacted>


STDERR
waiting for lock on working directory of /var/repo/8 held by process '112297' on host 'ec6149cd0a0b/f000037a'
abort: working directory of /var/repo/8: timed out waiting for lock held by 'ec6149cd0a0b/f000037a:112297'

Unfortunately, I don't really see a backtrace there. When I run bin/repository pull R8 from the command line, I see this error:

[2023-04-26 06:14:15] EXCEPTION: (Exception) Command failed with error #255!
COMMAND
hg --config ui.ssh=/var/www/phorge/phorge/bin/ssh-connect pull -- '********'

STDOUT
pulling from https://<redacted>


STDERR
waiting for lock on working directory of /var/repo/8 held by process '112297' on host 'ec6149cd0a0b/f000037a'
abort: working directory of /var/repo/8: timed out waiting for lock held by 'ec6149cd0a0b/f000037a:112297'
 at [<phorge>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:740]
arcanist(head=ca5f5cd1522c712f4bdb890ad49e993a9b51417c, ref.master=ca5f5cd1522c), phorge(head=1ba5c8c260950ce8bc77f261b41aa82ad5d776e2, ref.master=90f9da643d16)
  #0 PhabricatorRepositoryPullEngine::executeMercurialUpdate() called at [<phorge>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:134]
  #1 PhabricatorRepositoryPullEngine::pullRepositoryWithLock() called at [<phorge>/src/applications/repository/engine/PhabricatorRepositoryPullEngine.php:40]
  #2 PhabricatorRepositoryPullEngine::pullRepository() called at [<phorge>/src/applications/repository/management/PhabricatorRepositoryManagementPullWorkflow.php:50]
  #3 PhabricatorRepositoryManagementPullWorkflow::execute(PhutilArgumentParser) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:492]
  #4 PhutilArgumentParser::parseWorkflowsFull(array) called at [<arcanist>/src/parser/argument/PhutilArgumentParser.php:377]
  #5 PhutilArgumentParser::parseWorkflows(array) called at [<phorge>/scripts/repository/manage_repositories.php:22]

I did try to read through the code involved in that backtrace, and in PhabricatorRepositoryPullEngine::executeMercurialUpdate(), a future is created as I described above: Inside the repository's base class, which uses getEffectiveCopyTimeLimit() to determine the timeout.

I'm not sure where the buildUpdateFuture() function you linked to is used, but in the stack trace that I see (see previous comment), the future is buit inside executeMercurialUpdate(…), which calls $repository->getRemoteCommandFuture(), which boils down to the call newRemoteCommandEngine(…)->newFuture(); in the repo base class.

The newRemoteCommandEngine() calls returns a DiffusionCommandEngine, which uses getEffectiveCopyTimeLimit() in the newFuture() method.

I think the future() I linked for literally calls ./bin/repository update ..., and sets the timeout on that.
That invokes the future() you found, which calls hg pull, and adds another timeout.

I'm going to guess the dual-timeout is a bug (https://secure.phabricator.com/D19235 adds the 15 minutes one; D19831 makes it configurable), as in, it wasn't supposed to apply to this particular case.

I'll answer your original question as an "answer", but the tldr is "go ahead" and "manually update database".

Want to file the ticket?

From the exception

STDERR
waiting for lock on working directory of /var/repo/8 held by process '112297' on host 'ec6149cd0a0b/f000037a'

It sounds like another hg process is operating on that repo at the time.

Assuming this as Resolved but feel free to attach other details in the question itself and reopen :)

Answers

avivey
Updated 360 Days Ago
Actions

Looks like it was made git-only because it was only supposed to be used in the context of Drydock, which is git-only. I think having it effect the actual initial clone is a bug (See Comments).

In the spirit of T8227, the cleaner solution would be to fix the bug, leave the config git-only, and add some text explaining why it's git-only.

You can edit the detail in the database by directly editing the DB entry. It should be just a JSON blob in the phabricator_repository.repository table.

Bonus: You can also just manually do the clone by running the right hg pull command as the right user.

New Answer

Answer

This question has been marked as closed, but you can still leave a new answer.

How to set the Pull Time Limit for Mercurial repos?Closed, ResolvedPublicActions

Details

Event Timeline

Answers

aviveyUpdated 360 Days AgoActions

Event Timeline

New Answer

Answer

How to set the Pull Time Limit for Mercurial repos?
Closed, ResolvedPublic
Actions

avivey
Updated 360 Days Ago
Actions