Diffusion User Guide: Repository Hosting

Phorge Administrator and User Documentation (Application User Guides)

Guide to configuring Phorge repository hosting.

Overview

Phorge can host repositories and provide authenticated read and write access to them over HTTP and SSH. This document describes how to configure repository hosting.

Understanding Supported Protocols

Phorge supports hosting over these protocols:

All supported protocols handle reads (pull/checkout/clone) and writes (push/commit). Of the two protocols, SSH is generally more robust, secure and performant, but HTTP is easier to set up and supports anonymous access.

Each repository can be configured individually, and you can use either protocol, or both, or a mixture across different repositories.

SSH is recommended unless you need anonymous access, or are not able to configure it for technical reasons.

Creating System User Accounts

Phorge uses two system user accounts, plus a third account if you configure SSH access. This section will guide you through creating and configuring them. These are system user accounts on the machine Phorge runs on, not Phorge user accounts.

The system accounts Phorge uses are:

• The user the webserver runs as. We'll call this www-user.
• The user the daemons run as. We'll call this daemon-user. This user is the only user which will interact with the repositories directly. Other accounts will sudo to this account in order to perform repository operations.
• The user that humans will connect over SSH as. We'll call this vcs-user. If you do not plan to make repositories available over SSH, you do not need to create or configure this user.

To create these users:

• Create a www-user if one does not already exist. In most cases, this user will already exist and you just need to identify which user it is. Run your webserver as this user.
• Create a daemon-user if one does not already exist (you can call this user whatever you want, or use an existing account). Below, you'll configure the daemons to start as this user.
• Create a vcs-user if one does not already exist and you plan to set up SSH. When users clone repositories, they will use a URI like vcs-user@phorge.yourcompany.com, so common names for this user are git or hg.

Continue below to configure these accounts.

Configuring Phorge

Now that you have created or identified these accounts, update the Phorge configuration to specify them.

First, set phd.user to the daemon-user:

phorge/ $ ./bin/config set phd.user daemon-user

Restart the daemons to make sure this configuration works properly. They should start as the correct user automatically.

If you're using a vcs-user for SSH, you should also configure that:

phorge/ $ ./bin/config set diffusion.ssh-user vcs-user

Next, you'll set up sudo permissions so these users can interact with one another.

Configuring Sudo

The www-user and vcs-user need to be able to sudo as the daemon-user so they can interact with repositories.

To grant them access, edit the sudo system configuration. On many systems, you will do this by modifying the /etc/sudoers file using visudo or sudoedit. In some cases, you may add a new file to /etc/sudoers.d instead.

To give a user account sudo access to run a list of binaries, add a line like this to the configuration file (this example would grant vcs-user permission to run ls as daemon-user):

vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/bin/ls

The www-user needs to be able to run these binaries as the daemon-user:

• git (if using Git)
• git-http-backend (if using Git)
• hg (if using Mercurial)
• ssh (if configuring clusters)

If you plan to use SSH, the vcs-user needs to be able to run these binaries as the daemon-user:

• git (if using Git)
• git-upload-pack (if using Git)
• git-receive-pack (if using Git)
• hg (if using Mercurial)
• svnserve (if using Subversion)
• ssh (if configuring clusters)

Identify the full paths to all of these binaries on your system and add the appropriate permissions to the sudo configuration.

Normally, you'll add two lines that look something like this:

www-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/x, /path/to/y, ...
vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/x, /path/to/y, ...

This is just a template. In the real configuration file, you need to:

• Replace www-user, daemon-user and vcs-user with the correct usernames for your system.
• List every binary that these users need access to, as described above.
• Make sure each binary path is the full path to the correct binary location on your system.

Before continuing, look for this line in your sudo configuration:

Defaults    requiretty

If it's present, comment it out by putting a # at the beginning of the line. With this option enabled, VCS SSH sessions won't be able to use sudo.

Additional SSH User Configuration

If you're planning to use SSH, you should also edit /etc/passwd and /etc/shadow to make sure the vcs-user account is set up correctly.

/etc/shadow: Open /etc/shadow and find the line for the vcs-user account.

The second field (which is the password field) must not be set to !!. This value will prevent login.

If you have usermod on your system, you can adjust this value with:

$ sudo usermod -p NP vcs-user

If you do not have usermod, carefully edit the file and set the field value to NP ("no password") instead of !!.

/etc/passwd: Open /etc/passwd and find the line for the vcs-user account.

The last field (which is the login shell) must be set to a real shell. If it is set to something like /bin/false, then sshd will not be able to execute commands.

If you have usermod on your system, you can adjust this value with:

$ sudo usermod -s /bin/sh vcs-user

If you do not have usermod, carefully edit the file and change the field to point at a real shell, usually /bin/sh.

Configuring HTTP

If you plan to serve repositories over authenticated HTTP, you need to set diffusion.allow-http-auth in Config. If you don't plan to serve repositories over HTTP (or plan to use only anonymous HTTP) you can leave this setting disabled.

If you plan to use authenticated HTTP, you (and all other users) also need to configure a VCS password for your account in Settings → VCS Password.

Your VCS password must be a different password than your main Phorge password because VCS passwords are very easy to accidentally disclose. They are often stored in plaintext in world-readable files, observable in ps output, and present in command output and logs. We strongly encourage you to use SSH instead of HTTP to authenticate access to repositories.

Otherwise, if you've configured system accounts above, you're all set. No additional server configuration is required to make HTTP work. You should now be able to fetch and push repositories over HTTP. See "Cloning a Repository" below for more details.

If you're having trouble, see "Troubleshooting HTTP" below.

Configuring SSH

SSH access requires some additional setup. You will configure and run a second, restricted copy of sshd on the machine, on a different port from the standard sshd. This special copy of sshd will serve repository requests and provide other Phorge SSH services.

Before continuing, you must choose a strategy for which port each copy of sshd will run on. The next section lays out various approaches.

SSHD Port Assignment

The normal sshd that lets you administrate the host and the special sshd which serves repositories can't run on the same port. In particular, only one of them can run on port 22, which will make it a bit inconvenient to access the other one.

These instructions will walk you through configuring the alternate sshd on port 2222. This is easy to configure, but if you run the service on this port users will clone and push to URIs like ssh://git@host.com:2222/, which is a little ugly.

There are several different approaches you can use to mitigate or eliminate this problem.

Run on Port 2222: You can do nothing, and just run the repository sshd on port 2222 and accept the explicit port in the URIs. This is the simplest approach, and you can always start here and clean things up later if you grow tired of dealing with the port number.

Use a Load Balancer: You can configure a load balancer in front of the host and have it forward TCP traffic on port 22 to port 2222. Then users can clone from ssh://git@host.com/ without an explicit port number and you don't need to do anything else.

This may be very easy to set up, particularly if you are hosted in AWS, and is often the simplest and cleanest approach.

Swap Ports: You can move the administrative sshd to a new port, then run Phorge sshd on port 22. This is somewhat complicated and can be a bit risky if you make a mistake. See "Moving the sshd Port" below for help.

Change Client Config: You can run on a nonstandard port, but configure SSH on the client side so that ssh automatically defaults to the correct port when connecting to the host. To do this, add a section like this to your ~/.ssh/config:

Host phorge.corporation.com
  Port 2222

(If you want, you can also add a default User.)

Command line tools like ssh, git and hg will now default to port 2222 when connecting to this host.

A downside to this approach is that your users will each need to set up their ~/.ssh/config files individually.

This file also allows you to define short names for hosts using the Host and HostName options. If you choose to do this, be aware that Phorge uses remote/clone URIs to figure out which repository it is operating in, but can not resolve host aliases defined in your ssh config. If you create host aliases they may break some features related to repository identification.

If you use this approach, you will also need to specify a port explicitly when connecting to administrate the host. Any unit tests or other build automation will also need to be configured or use explicit port numbers.

Port Multiplexing: If you have hardware access, you can power down the host and find the network I/O pins on the motherboard (for onboard networking) or network card.

Carefully strip and solder a short piece of copper wire between the pins for the external interface 22 and internal 2222, so the external interface can receive traffic for both services.

(Make sure not to desolder the existing connection between external 22 and internal 22 or you won't be able to connect normally to administrate the host.)

The obvious downside to this approach is that it requires physical access to the machine, so it won't work if you're hosted on a cloud provider.

SSHD Setup

Now that you've decided how you'll handle port assignment, you're ready to continue sshd setup.

If you plan to connect to a port other than 22, you should set this port as diffusion.ssh-port in your Phorge config:

$ ./bin/config set diffusion.ssh-port 2222

This port is not special, and you are free to choose a different port, provided you make the appropriate configuration adjustment below.

Configure and Start Phorge SSHD: Now, you'll configure and start a copy of sshd which will serve Phorge services, including repositories, over SSH.

This instance will use a special locked-down configuration that uses Phorge to handle authentication and command execution.

There are three major steps:

• Create a phorge-ssh-hook.sh file.
• Create a `sshd_phorge config file.
• Start a copy of sshd using the new configuration.

Create phorge-ssh-hook.sh: Copy the template in phorge/resources/sshd/phorge-ssh-hook.sh to somewhere like /usr/libexec/phorge-ssh-hook.sh and edit it to have the correct settings.

Both the script itself and the parent directory the script resides in must be owned by root, and the script must have 755 permissions:

$ sudo chown root /path/to/somewhere/
$ sudo chown root /path/to/somewhere/phorge-ssh-hook.sh
$ sudo chmod 755  /path/to/somewhere/phorge-ssh-hook.sh

If you don't do this, sshd will refuse to execute the hook.

Create sshd_config for Phorge: Copy the template in phorge/resources/sshd/sshd_config.phorge.example to somewhere like /etc/ssh/sshd_config.phorge.

Open the file and edit the AuthorizedKeysCommand, AuthorizedKeysCommandUser, and AllowUsers settings to be correct for your system.

This configuration file also specifies the Port the service should run on. If you intend to run on a non-default port, adjust it now.

Start SSHD: Now, start the Phorge sshd:

sudo /path/to/sshd -f /path/to/sshd_config.phorge

If you did everything correctly, you should be able to run this command:

$ echo {} | ssh vcs-user@phorge.yourcompany.com conduit conduit.ping

...and get a response like this:

{"result":"phorge.yourcompany.com","error_code":null,"error_info":null}

If you get an authentication error, make sure you added your public key in Settings → SSH Public Keys. If you're having trouble, check the troubleshooting section below.

Authentication Over SSH

To authenticate over SSH, users should add their public keys under Settings → SSH Public Keys.

Cloning a Repository

If you've already set up a hosted repository, you can try cloning it now. To do this, browse to the repository's main screen in Diffusion. You should see clone commands at the top of the page.

To clone the repository, just run the appropriate command.

If you don't see the commands or running them doesn't work, see below for tips on troubleshooting.

Troubleshooting HTTP

Some general tips for troubleshooting problems with HTTP:

• Make sure diffusion.allow-http-auth is enabled in your Phorge config.
• Make sure HTTP serving is enabled for the repository you're trying to clone. You can find this in Edit Repository → Hosting.
• Make sure you've configured a VCS password. This is separate from your main account password. You can configure this in Settings → VCS Password.
• Make sure the main repository screen in Diffusion shows a clone/checkout command for HTTP. If it doesn't, something above isn't set up correctly: double-check your configuration. You should see a svn checkout http://..., git clone http://... or hg clone http://... command. Run that command verbatim to clone the repository.

If you're using Git, using GIT_CURL_VERBOSE may help assess login failures. To do so, specify it on the command line before the git clone command, like this:

$ GIT_CURL_VERBOSE=1 git clone ...

This will make git print out a lot more information. Particularly, the line with the HTTP response is likely to be useful:

< HTTP/1.1 403 Invalid credentials.

In many cases, this can give you more information about what's wrong.

Troubleshooting SSH

Some general tips for troubleshooting problems with SSH:

• Check that you've configured diffusion.ssh-user.
• Check that you've configured phd.user.
• Make sure SSH serving is enabled for the repository you're trying to clone. You can change this setting from a main repository screen in Diffusion by Edit Repository → Edit Hosting → Host Repository on Phabricator → Save and Continue → SSH Read Only or Read/Write → Save Changes.
• Make sure you've added an SSH public key to your account. You can do this in Settings → SSH Public Keys.
• Make sure the main repository screen in Diffusion shows a clone/checkout command for SSH. If it doesn't, something above isn't set up correctly. You should see an svn checkout svn+ssh://..., git clone ssh://... or hg clone ssh://... command. Run that command verbatim to clone the repository.
• Check your phorge-ssh-hook.sh file for proper settings.
• Check your sshd_config.phorge file for proper settings.

To troubleshoot SSH setup: connect to the server with ssh, without running a command. You may need to use the -T flag, and will need to use -p if you are running on a nonstandard port. You should see a message like this one:

$ ssh -T -p 2222 vcs-user@phorge.yourcompany.com
phorge-ssh-exec: Welcome to Phorge.

You are logged in as alincoln.

You haven't specified a command to run. This means you're requesting an
interactive shell, but Phorge does not provide an interactive shell over
SSH.

Usually, you should run a command like `git clone` or `hg push` rather than
connecting directly with SSH.

Supported commands are: conduit, git-receive-pack, git-upload-pack, hg,
svnserve.

If you see this message, all your SSH stuff is configured correctly. If you get a login shell instead, you've missed some major setup step: review the documentation above. If you get some other sort of error, double check these settings:

• You're connecting as the vcs-user.
• The vcs-user has NP in /etc/shadow.
• The vcs-user has /bin/sh or some other valid shell in /etc/passwd.
• Your SSH private key is correct, and you've added the corresponding public key to Phorge in the Settings panel.

If you can get this far, but can't execute VCS commands like git clone, there is probably an issue with your sudoers configuration. Check:

• Your sudoers file is set up as instructed above.
• You've commented out Defaults requiretty in sudoers.
• You don't have multiple copies of the VCS binaries (like git-upload-pack) on your system. You may have granted sudo access to one, while the VCS user is trying to run a different one.
• You've configured phd.user.
• The phd.user has read and write access to the repositories.

It may also be helpful to run sshd in debug mode:

$ /path/to/sshd -d -d -d -f /path/to/sshd_config.phorge

This will run it in the foreground and emit a large amount of debugging information when you connect to it.

Finally, you can usually test that sudoers is configured correctly by doing something like this:

$ su vcs-user
$ sudo -E -n -u daemon-user -- /path/to/some/vcs-binary --help

That will try to run the binary via sudo in a manner similar to the way that Phorge will run it. This can give you better error messages about issues with sudoers configuration.

Miscellaneous Troubleshooting

• If you're getting an error about svnlook not being found, add the path where svnlook is located to the Phorge configuration environment.append-paths (even if it already appears in PATH). This issue is caused by SVN wiping the environment (including PATH) when invoking commit hooks.

Moving the sshd Port

If you want to move the standard (administrative) sshd to a different port to make Phorge repository URIs cleaner, this section has some tips.

This is optional, and it is normally easier to do this by putting a load balancer in front of Phorge and having it accept TCP traffic on port 22 and forward it to some other port.

When moving sshd, be careful when editing the configuration. If you get it wrong, you may lock yourself out of the machine. Restarting sshd generally will not interrupt existing connections, but you should exercise caution. Two strategies you can use to mitigate this risk are: smoke-test configuration by starting a second sshd; and use a screen session which automatically repairs configuration unless stopped.

To smoke-test a configuration, just start another sshd using the -f flag:

sudo /path/to/sshd -f /path/to/config_file.edited

You can then connect and make sure the edited config file is valid before replacing your primary configuration file.

To automatically repair configuration, start a screen session with a command like this in it:

sleep 60 ; mv sshd_config.good sshd_config ; /etc/init.d/sshd restart

The specific command may vary for your system, but the general idea is to have the machine automatically restore configuration after some period of time if you don't stop it. If you lock yourself out, this can fix things automatically.

Now that you're ready to edit your configuration, open up your sshd config (often /etc/ssh/sshd_config) and change the Port setting to some other port, like 222 (you can choose any port other than 22).

Port 222

Very carefully, restart sshd. Verify that you can connect on the new port:

ssh -p 222 ...

Now you can move the Phorge sshd to port 22, then adjust the value for diffusion.ssh-port in your Phorge configuration.

You can set up and enable this systemd unit to start the second sshd daemon on every reboot:

[Unit]
Description=Phorge sshd
Documentation=https://we.phorge.it/book/phorge/article/diffusion_hosting/#sshd-setup
After=network.target auditd.service

[Service]
ExecStartPre=/usr/sbin/sshd -t -f /path/to/config_file.edited
ExecStart=/usr/sbin/sshd -f /path/to/config_file.edited
ExecReload=/usr/sbin/sshd -t -f /path/to/config_file.edited
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=phorge-sshd.service

No Direct Pushes

You may get an error about "No Direct Pushes" when trying to push. This means you are pushing directly to the repository instead of pushing through Phorge. This is not supported: writes to hosted repositories must go through Phorge so it can perform authentication, enforce permissions, write logs, proxy requests, apply rewriting, etc.

One way to do a direct push by mistake is to use a file:/// URI to interact with the repository from the same machine. This is not supported. Instead, use one of the repository URIs provided in the web interface, even if you're working on the same machine.

Another way to do a direct push is to misconfigure SSH (or not configure it at all) so that none of the logic described above runs and you just connect normally as a system user. In this case, the ssh test described above will fail (you'll get a command prompt when you connect, instead of the message you are supposed to get, as described above).

If you encounter this error: make sure you're using a remote URI given to you by Diffusion in the web interface, then run through the troubleshooting steps above carefully.

Sometimes users encounter this problem because they skip this whole document assuming they don't need to configure anything. This will not work, and you MUST configure things as described above for hosted repositories to work.

The technical reason this error occurs is that the PHABRICATOR_USER variable is not defined in the environment when commit hooks run. This variable is set by Phorge when a request passes through the authentication layer that this document provides instructions for configuring. Its absence indicates that the request did not pass through Phorge.

Next Steps

Once hosted repositories are set up:

• learn about commit hooks with Diffusion User Guide: Commit Hooks.