Changeset View
Changeset View
Standalone View
Standalone View
src/docs/user/field/repository_imports.diviner
@title Troubleshooting Repository Imports | @title Troubleshooting Repository Imports | ||||
@group fieldmanual | @group fieldmanual | ||||
Guide to the troubleshooting repositories which import incompletely. | Guide to the troubleshooting repositories which import incompletely. | ||||
Overview | Overview | ||||
======== | ======== | ||||
When you first import an external source code repository (or push new commits to | When you first import an external source code repository (or push new commits to | ||||
a hosted repository), Phabricator imports those commits in the background. | a hosted repository), Phorge imports those commits in the background. | ||||
While a repository is initially importing, some features won't work. While | While a repository is initially importing, some features won't work. While | ||||
individual commits are importing, some of their metadata won't be available in | individual commits are importing, some of their metadata won't be available in | ||||
the web UI. | the web UI. | ||||
Sometimes, the import process may hang or fail to complete. This document can | Sometimes, the import process may hang or fail to complete. This document can | ||||
help you understand the import process and troubleshoot problems with it. | help you understand the import process and troubleshoot problems with it. | ||||
Understanding the Import Pipeline | Understanding the Import Pipeline | ||||
================================= | ================================= | ||||
Phabricator first performs commit discovery on repositories. This examines | Phorge first performs commit discovery on repositories. This examines | ||||
a repository and identifies all the commits in it at a very shallow level, | a repository and identifies all the commits in it at a very shallow level, | ||||
then creates stub objects for them. These stub objects primarily serve to | then creates stub objects for them. These stub objects primarily serve to | ||||
assign various internal IDs to each commit. | assign various internal IDs to each commit. | ||||
Commit discovery occurs in the update phase, and you can learn more about | Commit discovery occurs in the update phase, and you can learn more about | ||||
updates in @{article:Diffusion User Guide: Repository Updates}. | updates in @{article:Diffusion User Guide: Repository Updates}. | ||||
After commits are discovered, background tasks are queued to actually import | After commits are discovered, background tasks are queued to actually import | ||||
Show All 17 Lines | |||||
========================= | ========================= | ||||
There are a few major pieces of information you can look at to understand where | There are a few major pieces of information you can look at to understand where | ||||
the import process is stuck. | the import process is stuck. | ||||
First, to identify which commits have missing import steps, run this command: | First, to identify which commits have missing import steps, run this command: | ||||
``` | ``` | ||||
phabricator/ $ ./bin/repository importing rXYZ | phorge/ $ ./bin/repository importing rXYZ | ||||
``` | ``` | ||||
That will show what work remains to be done. Each line shows a commit which | That will show what work remains to be done. Each line shows a commit which | ||||
is discovered but not imported, and the import steps that are remaining for | is discovered but not imported, and the import steps that are remaining for | ||||
that commit. Generally, the commit is stuck on the first step in the list. | that commit. Generally, the commit is stuck on the first step in the list. | ||||
Second, load the Daemon Console (at `/daemon/` in the web UI). This will show | Second, load the Daemon Console (at `/daemon/` in the web UI). This will show | ||||
what work is currently being done and waiting to complete. The most important | what work is currently being done and waiting to complete. The most important | ||||
sections are "Queued Tasks" (work waiting in queue) and "Leased Tasks" | sections are "Queued Tasks" (work waiting in queue) and "Leased Tasks" | ||||
(work currently being done). | (work currently being done). | ||||
Third, run this command to look at the daemon logs: | Third, run this command to look at the daemon logs: | ||||
``` | ``` | ||||
phabricator/ $ ./bin/phd log | phorge/ $ ./bin/phd log | ||||
``` | ``` | ||||
This can show you any errors the daemons have encountered recently. | This can show you any errors the daemons have encountered recently. | ||||
The next sections will walk through how to use this information to understand | The next sections will walk through how to use this information to understand | ||||
and resolve the issue. | and resolve the issue. | ||||
Show All 18 Lines | |||||
they're failing to import. | they're failing to import. | ||||
These failures are the easiest to identify and understand, and can often be | These failures are the easiest to identify and understand, and can often be | ||||
resolved quickly. Choose some failing commit from the output of `bin/repository | resolved quickly. Choose some failing commit from the output of `bin/repository | ||||
importing` and use this command to re-run any missing steps manually in the | importing` and use this command to re-run any missing steps manually in the | ||||
foreground: | foreground: | ||||
``` | ``` | ||||
phabricator/ $ ./bin/repository reparse --importing --trace rXYZabcdef012... | phorge/ $ ./bin/repository reparse --importing --trace rXYZabcdef012... | ||||
``` | ``` | ||||
This command is always safe to run, no matter what the actual root cause of | This command is always safe to run, no matter what the actual root cause of | ||||
the problem is. | the problem is. | ||||
If this fails with an error, you've likely identified a problem with | If this fails with an error, you've likely identified a problem with | ||||
Phabricator. Collect as much information as you can about what makes the commit | Phorge. Collect as much information as you can about what makes the commit | ||||
special and file a bug in the upstream by following the instructions in | special and file a bug in the upstream by following the instructions in | ||||
@{article:Contributing Bug Reports}. | @{article:Contributing Bug Reports}. | ||||
If the commit imports cleanly, this is more likely to be caused by some other | If the commit imports cleanly, this is more likely to be caused by some other | ||||
issue. | issue. | ||||
Handling Temporary Failures | Handling Temporary Failures | ||||
=========================== | =========================== | ||||
Some commits may temporarily fail to import: perhaps the network or services | Some commits may temporarily fail to import: perhaps the network or services | ||||
may have briefly been down, or some configuration wasn't quite right, or the | may have briefly been down, or some configuration wasn't quite right, or the | ||||
daemons were killed halfway through the work. | daemons were killed halfway through the work. | ||||
These commits will retry eventually and usually succeed, but some of the retry | These commits will retry eventually and usually succeed, but some of the retry | ||||
time limits are very conservative (up to 24 hours) and you might not want to | time limits are very conservative (up to 24 hours) and you might not want to | ||||
wait that long. | wait that long. | ||||
In the Daemon console, temporarily failures usually look like tasks in the | In the Daemon console, temporarily failures usually look like tasks in the | ||||
"Leased Tasks" column with a large "Expires" value but a low "Failures" count | "Leased Tasks" column with a large "Expires" value but a low "Failures" count | ||||
(usually 0 or 1). The "Expires" column is showing how long Phabricator is | (usually 0 or 1). The "Expires" column is showing how long Phorge is | ||||
waiting to retry these tasks. | waiting to retry these tasks. | ||||
In the daemon log, these temporary failures might have created log entries, but | In the daemon log, these temporary failures might have created log entries, but | ||||
might also not have. For example, if the failure was rooted in a network issue | might also not have. For example, if the failure was rooted in a network issue | ||||
that probably will create a log entry, but if the failure was rooted in the | that probably will create a log entry, but if the failure was rooted in the | ||||
daemons being abruptly killed that may not create a log entry. | daemons being abruptly killed that may not create a log entry. | ||||
You can follow the instructions from "Handling Permanent Failures" above to | You can follow the instructions from "Handling Permanent Failures" above to | ||||
reparse steps individually to look for an error that represents a root cause, | reparse steps individually to look for an error that represents a root cause, | ||||
but sometimes this can happen because of some transient issue which won't be | but sometimes this can happen because of some transient issue which won't be | ||||
identifiable. | identifiable. | ||||
The easiest way to fix this is to restart the daemons. When you restart | The easiest way to fix this is to restart the daemons. When you restart | ||||
daemons, all task leases are immediately expired, so any tasks waiting for a | daemons, all task leases are immediately expired, so any tasks waiting for a | ||||
long time will run right away: | long time will run right away: | ||||
``` | ``` | ||||
phabricator/ $ ./bin/phd restart | phorge/ $ ./bin/phd restart | ||||
``` | ``` | ||||
This command is always safe to run, no matter what the actual root cause of | This command is always safe to run, no matter what the actual root cause of | ||||
the problem is. | the problem is. | ||||
After restarting the daemons, any pending tasks should be able to retry | After restarting the daemons, any pending tasks should be able to retry | ||||
immediately. | immediately. | ||||
For more information on managing the daemons, see | For more information on managing the daemons, see | ||||
@{article:Managing Daemons with phd}. | @{article:Managing Daemons with phd}. | ||||
Forced Parsing | Forced Parsing | ||||
============== | ============== | ||||
In rare cases, the actual tasks may be lost from the task queue. Usually, they | In rare cases, the actual tasks may be lost from the task queue. Usually, they | ||||
have been stolen by gremlins or spirited away by ghosts, or someone may have | have been stolen by gremlins or spirited away by ghosts, or someone may have | ||||
been too ambitious with running manual SQL commands and deleted a bunch of | been too ambitious with running manual SQL commands and deleted a bunch of | ||||
extra things they shouldn't have. | extra things they shouldn't have. | ||||
There is no normal set of conditions under which this should occur, but you can | There is no normal set of conditions under which this should occur, but you can | ||||
force Phabricator to re-queue the tasks to recover from it if it does occur. | force Phorge to re-queue the tasks to recover from it if it does occur. | ||||
This will look like missing steps in `repository importing`, but nothing in the | This will look like missing steps in `repository importing`, but nothing in the | ||||
"Queued Tasks" or "Leased Tasks" sections of the daemon console. The daemon | "Queued Tasks" or "Leased Tasks" sections of the daemon console. The daemon | ||||
logs will also be empty, since the tasks have vanished. | logs will also be empty, since the tasks have vanished. | ||||
To re-queue parse tasks for a repository, run this command, which will queue | To re-queue parse tasks for a repository, run this command, which will queue | ||||
up all of the missing work in `repository importing`: | up all of the missing work in `repository importing`: | ||||
``` | ``` | ||||
phabricator/ $ ./bin/repository reparse --importing --all rXYZ | phorge/ $ ./bin/repository reparse --importing --all rXYZ | ||||
``` | ``` | ||||
This command may cause duplicate work to occur if you have misdiagnosed the | This command may cause duplicate work to occur if you have misdiagnosed the | ||||
problem and the tasks aren't actually lost. For example, it could queue a | problem and the tasks aren't actually lost. For example, it could queue a | ||||
second task to perform publishing, which could cause Phabricator to send a | second task to perform publishing, which could cause Phorge to send a | ||||
second copy of email about the commit. Other than that, it is safe to run even | second copy of email about the commit. Other than that, it is safe to run even | ||||
if this isn't the problem. | if this isn't the problem. | ||||
After running this command, you should see tasks in "Queued Tasks" and "Leased | After running this command, you should see tasks in "Queued Tasks" and "Leased | ||||
Tasks" in the console which correspond to the commits in `repository | Tasks" in the console which correspond to the commits in `repository | ||||
importing`, and progress should resume. | importing`, and progress should resume. | ||||
Show All 11 Lines | |||||
publishing of notifications and email, so if you flag a repository as imported | publishing of notifications and email, so if you flag a repository as imported | ||||
but it still has a lot of work queued, it may send an enormous amount of email | but it still has a lot of work queued, it may send an enormous amount of email | ||||
as that work completes. | as that work completes. | ||||
To mark a repository as imported even though it really isn't, run this | To mark a repository as imported even though it really isn't, run this | ||||
command: | command: | ||||
``` | ``` | ||||
phabricator/ $ ./bin/repository mark-imported rXYZ | phorge/ $ ./bin/repository mark-imported rXYZ | ||||
``` | ``` | ||||
If you do this by mistake, you can reverse it later by using the | If you do this by mistake, you can reverse it later by using the | ||||
`--mark-not-imported` flag. | `--mark-not-imported` flag. | ||||
General Tips | General Tips | ||||
============ | ============ | ||||
Broadly, `bin/repository` contains several useful debugging commands which | Broadly, `bin/repository` contains several useful debugging commands which | ||||
let you figure out where failures are occurring. You can add the `--trace` flag | let you figure out where failures are occurring. You can add the `--trace` flag | ||||
to any command to get more details about what it is doing. For any command, | to any command to get more details about what it is doing. For any command, | ||||
you can use `help` to learn more about what it does and which flag it takes: | you can use `help` to learn more about what it does and which flag it takes: | ||||
``` | ``` | ||||
phabricator/ $ bin/repository help <command> | phorge/ $ bin/repository help <command> | ||||
``` | ``` | ||||
In particular, you can use flags with the `repository reparse` command to | In particular, you can use flags with the `repository reparse` command to | ||||
manually run parse steps in the foreground, including re-running steps and | manually run parse steps in the foreground, including re-running steps and | ||||
running steps out of order. | running steps out of order. | ||||
Next Steps | Next Steps | ||||
========== | ========== | ||||
Continue by: | Continue by: | ||||
- returning to the @{article:Diffusion User Guide}. | - returning to the @{article:Diffusion User Guide}. |
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under Apache 2.0 or other open source licenses. · CC BY-SA 4.0 · Apache 2.0