Page MenuHomePhorge

Exception when importing Mercurial repository with non-UTF-8 characters in filenames
Closed, ObsoletePublic

Asked by tinloaf on Apr 24 2023, 14:21.

Details

Hi,

I'm trying to import a (really old…) Mercurial repository in 'observe' mode into a recent Phorge (cloned a week ago). The repository is large (~175k commits), and 60 of those commits fail to import. Not a bad ratio, but I still would like to get the repository into a 'fully imported' state. I know I can manually set it to 'imported', but I'm not sure what the consequences of that are, whether I should expect missing files etc. So, it would be optimal to get those remaining 60 commits to import.

Under /daemon/, I can see PhabricatorRepositoryMercurialCommitChangeParserWorker jobs with high failure counts. This is what the repository tool tells me:

root@ec6149cd0a0b:/var/www/phorge/phorge# ./bin/repository importing R3
R3:f60bb794a270 Change, Publish
R3:5eeae9954ae8 Change, Publish
R3:773043ec1398 Change, Publish
[…]

They are all standing on Change, Publish. Let's see what phd log says:

Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #13 PhutilDaemon::execute() called at [<phorge>/scripts/daemon/exec/exec_daemon.php:131]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000] [2023-04-24 14:14:05] EXCEPTION: (PhutilProxyException) Error while executing Task ID 644343. {>} (AphrontCharacterSetQueryException) Attempting to construct a query using a non-utf8 string when utf8 is expected. Use the `%B` conversion to escape binary strings data. at [<phorge>/src/infrastructure/storage/connection/mysql/AphrontBaseMySQLDatabaseConnection.php:418]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000] arcanist(head=master, ref.master=08dfffd5caf7), phorge(head=master, ref.master=b587865ce78a)
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #0 <#2> AphrontBaseMySQLDatabaseConnection::validateUTF8String(string) called at [<phorge>/src/infrastructure/storage/connection/mysql/AphrontMySQLiDatabaseConnection.php:12]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #1 <#2> AphrontMySQLiDatabaseConnection::escapeUTF8String(string) called at [<phorge>/src/infrastructure/storage/xsprintf/qsprintf.php:266]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #2 <#2> xsprintf_query(array, string, integer, string, integer) called at [<arcanist>/src/xsprintf/xsprintf.php:82]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #3 <#2> xsprintf(string, array, array) called at [<phorge>/src/infrastructure/storage/xsprintf/PhutilQueryString.php:31]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #4 <#2> PhutilQueryString::__construct(AphrontMySQLiDatabaseConnection, array) called at [<phorge>/src/infrastructure/storage/xsprintf/qsprintf.php:78]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #5 <#2> qsprintf(AphrontMySQLiDatabaseConnection, string, string, string) called at [<phorge>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:69]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #6 <#2> PhabricatorRepositoryCommitChangeParserWorker::lookupOrCreatePaths(array) called at [<phorge>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryMercurialCommitChangeParserWorker.php:255]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #7 <#2> PhabricatorRepositoryMercurialCommitChangeParserWorker::parseCommitChanges(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phorge>/src/applications/repository/worker/commitchangeparser/PhabricatorRepositoryCommitChangeParserWorker.php:36]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #8 <#2> PhabricatorRepositoryCommitChangeParserWorker::parseCommit(PhabricatorRepository, PhabricatorRepositoryCommit) called at [<phorge>/src/applications/repository/worker/PhabricatorRepositoryCommitParserWorker.php:72]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #9 <#2> PhabricatorRepositoryCommitParserWorker::doWork() called at [<phorge>/src/infrastructure/daemon/workers/PhabricatorWorker.php:124]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #10 <#2> PhabricatorWorker::executeTask() called at [<phorge>/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php:160]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #11 <#2> PhabricatorWorkerActiveTask::executeTask() called at [<phorge>/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php:22]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #12 PhabricatorTaskmasterDaemon::run() called at [<phorge>/src/infrastructure/daemon/PhutilDaemon.php:219]
Daemon 26 STDE [Mon, 24 Apr 2023 14:14:05 +0000]   #13 PhutilDaemon::execute() called at [<phorge>/scripts/daemon/exec/exec_daemon.php:131]
Daemon 26 FAIL [Mon, 24 Apr 2023 14:14:05 +0000] Process exited with error 255.

To my untrained eye this looks like it's trying to create a file path in the database and chokes on the encoding of the path. In fact, when I look at the problematic commits in my repository, I see that they contain files with German "Umlaute" ("ü", "ö", "ä", …) in their file names. I did not verify this for all 60 failed commits, but picked a couple of them at random. The commits are mostly from 1999 (did I mention the repo is old?). I assume that these file names are encoded in CP-1252, because this was Windows in the year 1999.

I'm not sure how to reproduce this with a test repository, because I'm honestly not sure how to create a non-utf-8 filename and force it into Mercurial in the year 2023. However, I see nothing else special about these changesets.

I will gladly open a bug report for this, but I don't think I have the necessary permissions.

Answers

New Answer

Answer

This question has been marked as closed, but you can still leave a new answer.