Something went wrong. Try again.
@recaptime-dev's working patches + fork for Phorge, a community fork of Phabricator. (Upstream dev and stable branches are at upstream/main and upstream/stable respectively.) hq.recaptime.dev/wiki/Phorge
phorge phabricator
Something went wrong. Try again.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661@title Diffusion User Guide: Repository Hosting@group userguideGuide to configuring Phorge repository hosting.Overview========Phorge can host repositories and provide authenticated read and writeaccess to them over HTTP and SSH. This document describes how to configurerepository hosting.Understanding Supported Protocols=================================Phorge supports hosting over these protocols:| VCS | SSH | HTTP ||-----|-----|------|| Git | Supported | Supported || Mercurial | Supported | Supported || Subversion | Supported | Not Supported |All supported protocols handle reads (pull/checkout/clone) and writes(push/commit). Of the two protocols, SSH is generally more robust, secure andperformant, but HTTP is easier to set up and supports anonymous access.| | SSH | HTTP || |-----|------|| Reads | Yes | Yes || Writes | Yes | Yes || Authenticated Access | Yes | Yes || Push Logs | Yes | Yes || Commit Hooks | Yes | Yes || Anonymous Access | No | Yes || Security | Better (Asymmetric Key) | Okay (Password) || Performance | Better | Okay || Setup | Hard | Easy |Each repository can be configured individually, and you can use eitherprotocol, or both, or a mixture across different repositories.SSH is recommended unless you need anonymous access, or are not able toconfigure it for technical reasons.Creating System User Accounts=============================Phorge uses two system user accounts, plus a third account if youconfigure SSH access. This section will guide you through creating andconfiguring them. These are system user accounts on the machine Phorgeruns on, not Phorge user accounts.The system accounts Phorge uses are: - The user the webserver runs as. We'll call this `www-user`. - The user the daemons run as. We'll call this `daemon-user`. This user is the only user which will interact with the repositories directly. Other accounts will `sudo` to this account in order to perform repository operations. - The user that humans will connect over SSH as. We'll call this `vcs-user`. If you do not plan to make repositories available over SSH, you do not need to create or configure this user.IMPORTANT: As a general service management philosophy, service users arevirtually always named after the service they are for. You are **strongly**encouraged to pick something other than `daemon-user` for running the daemons. Agood username might be `phd`.To create these users: - Create a `www-user` if one does not already exist. In most cases, this user will already exist and you just need to identify which user it is. Run your webserver as this user. - Create a `daemon-user` if one does not already exist (you can call this user whatever you want, or use an existing account). Below, you'll configure the daemons to start as this user. - Create a `vcs-user` if one does not already exist and you plan to set up SSH. When users clone repositories, they will use a URI like `vcs-user@phorge.yourcompany.com`, so common names for this user are `git` or `hg`.Continue below to configure these accounts.Configuring Phorge=======================Now that you have created or identified these accounts, update the Phorgeconfiguration to specify them.First, set `phd.user` to the `daemon-user`:```phorge/ $ ./bin/config set phd.user daemon-user```Restart the daemons to make sure this configuration works properly. They shouldstart as the correct user automatically.If you're using a `vcs-user` for SSH, you should also configure that:```phorge/ $ ./bin/config set diffusion.ssh-user vcs-user```Next, you'll set up `sudo` permissions so these users can interact with oneanother.Configuring Sudo================The `www-user` and `vcs-user` need to be able to `sudo` as the `daemon-user`so they can interact with repositories.To grant them access, edit the `sudo` system configuration. On many systems,you will do this by modifying the `/etc/sudoers` file using `visudo` or`sudoedit`. In some cases, you may add a new file to `/etc/sudoers.d` instead.To give a user account `sudo` access to run a list of binaries, add a line likethis to the configuration file (this example would grant `vcs-user` permissionto run `ls` as `daemon-user`):```vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/bin/ls```The `www-user` needs to be able to run these binaries as the `daemon-user`: - `git` (if using Git) - `git-http-backend` (if using Git) - `hg` (if using Mercurial) - `ssh` (if configuring clusters)If you plan to use SSH, the `vcs-user` needs to be able to run these binariesas the `daemon-user`: - `git` (if using Git) - `git-upload-pack` (if using Git) - `git-receive-pack` (if using Git) - `hg` (if using Mercurial) - `svnserve` (if using Subversion) - `ssh` (if configuring clusters)Identify the full paths to all of these binaries on your system and add theappropriate permissions to the `sudo` configuration.Normally, you'll add two lines that look something like this:```www-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/x, /path/to/y, ...vcs-user ALL=(daemon-user) SETENV: NOPASSWD: /path/to/x, /path/to/y, ...```This is just a template. In the real configuration file, you need to: - Replace `www-user`, `daemon-user` and `vcs-user` with the correct usernames for your system. - List every binary that these users need access to, as described above. - Make sure each binary path is the full path to the correct binary location on your system.Before continuing, look for this line in your `sudo` configuration: Defaults requirettyIf it's present, comment it out by putting a `#` at the beginning of the line.With this option enabled, VCS SSH sessions won't be able to use `sudo`.Additional SSH User Configuration=================================If you're planning to use SSH, you should also edit `/etc/passwd` and`/etc/shadow` to make sure the `vcs-user` account is set up correctly.**`/etc/shadow`**: Open `/etc/shadow` and find the line for the `vcs-user`account.The second field (which is the password field) must not be set to `!!`. Thisvalue will prevent login.If you have `usermod` on your system, you can adjust this value with:```$ sudo usermod -p NP vcs-user```If you do not have `usermod`, carefully edit the file and set the field valueto `NP` ("no password") instead of `!!`.**`/etc/passwd`**: Open `/etc/passwd` and find the line for the `vcs-user`account.The last field (which is the login shell) must be set to a real shell. If it isset to something like `/bin/false`, then `sshd` will not be able to executecommands.If you have `usermod` on your system, you can adjust this value with:```$ sudo usermod -s /bin/sh vcs-user```If you do not have `usermod`, carefully edit the file and change the fieldto point at a real shell, usually `/bin/sh`.Configuring HTTP================If you plan to serve repositories over authenticated HTTP, you need to set`diffusion.allow-http-auth` in Config. If you don't plan to serve repositoriesover HTTP (or plan to use only anonymous HTTP) you can leave this settingdisabled.If you plan to use authenticated HTTP, you (and all other users) also need toconfigure a VCS password for your account in {nav Settings > VCS Password}.Your VCS password must be a different password than your main Phorgepassword because VCS passwords are very easy to accidentally disclose. They areoften stored in plaintext in world-readable files, observable in `ps` output,and present in command output and logs. We strongly encourage you to use SSHinstead of HTTP to authenticate access to repositories.Otherwise, if you've configured system accounts above, you're all set. Noadditional server configuration is required to make HTTP work. You should nowbe able to fetch and push repositories over HTTP. See "Cloning a Repository"below for more details.If you're having trouble, see "Troubleshooting HTTP" below.Configuring SSH===============SSH access requires some additional setup. You will configure and run a second,restricted copy of `sshd` on the machine, on a different port from the standard`sshd`. This special copy of `sshd` will serve repository requests and provideother Phorge SSH services.NOTE: The Phorge `sshd` service **MUST** be 6.2 or newer, becausePhorge relies on the `AuthorizedKeysCommand` option.Before continuing, you must choose a strategy for which port each copy of`sshd` will run on. The next section lays out various approaches.SSHD Port Assignment====================The normal `sshd` that lets you administrate the host and the special `sshd`which serves repositories can't run on the same port. In particular, only oneof them can run on port `22`, which will make it a bit inconvenient to accessthe other one.These instructions will walk you through configuring the alternate `sshd` onport `2222`. This is easy to configure, but if you run the service on this portusers will clone and push to URIs like `ssh://git@host.com:2222/`, which is alittle ugly.There are several different approaches you can use to mitigate or eliminatethis problem.**Run on Port 2222**: You can do nothing, and just run the repository `sshd` onport `2222` and accept the explicit port in the URIs. This is the simplestapproach, and you can always start here and clean things up later if you growtired of dealing with the port number.**Use a Load Balancer**: You can configure a load balancer in front of the hostand have it forward TCP traffic on port `22` to port `2222`. Then users canclone from `ssh://git@host.com/` without an explicit port number and you don'tneed to do anything else.This may be very easy to set up, particularly if you are hosted in AWS, andis often the simplest and cleanest approach.**Swap Ports**: You can move the administrative `sshd` to a new port, then runPhorge `sshd` on port 22. This is somewhat complicated and can be a bitrisky if you make a mistake. See "Moving the sshd Port" below for help.**Change Client Config**: You can run on a nonstandard port, but configure SSHon the client side so that `ssh` automatically defaults to the correct portwhen connecting to the host. To do this, add a section like this to your`~/.ssh/config`:```Host phorge.corporation.com Port 2222```(If you want, you can also add a default `User`.)Command line tools like `ssh`, `git` and `hg` will now default to port`2222` when connecting to this host.A downside to this approach is that your users will each need to set up their`~/.ssh/config` files individually.This file also allows you to define short names for hosts using the `Host` and`HostName` options. If you choose to do this, be aware that Phorge usesremote/clone URIs to figure out which repository it is operating in, but cannot resolve host aliases defined in your `ssh` config. If you create hostaliases they may break some features related to repository identification.If you use this approach, you will also need to specify a port explicitly whenconnecting to administrate the host. Any unit tests or other build automationwill also need to be configured or use explicit port numbers.**Port Multiplexing**: If you have hardware access, you can power down the hostand find the network I/O pins on the motherboard (for onboard networking) ornetwork card.Carefully strip and solder a short piece of copper wire between the pins forthe external interface `22` and internal `2222`, so the external interface canreceive traffic for both services.(Make sure not to desolder the existing connection between external `22` andinternal `22` or you won't be able to connect normally to administrate thehost.)The obvious downside to this approach is that it requires physical access tothe machine, so it won't work if you're hosted on a cloud provider.SSHD Setup==========Now that you've decided how you'll handle port assignment, you're ready tocontinue `sshd` setup.If you plan to connect to a port other than `22`, you should set this portas `diffusion.ssh-port` in your Phorge config:```$ ./bin/config set diffusion.ssh-port 2222```This port is not special, and you are free to choose a different port, providedyou make the appropriate configuration adjustment below.**Configure and Start Phorge SSHD**: Now, you'll configure and start acopy of `sshd` which will serve Phorge services, including repositories,over SSH.This instance will use a special locked-down configuration that usesPhorge to handle authentication and command execution.There are three major steps: - Create a `phorge-ssh-hook.sh` file. - Create a `sshd_phorge config file. - Start a copy of `sshd` using the new configuration.**Create `phorge-ssh-hook.sh`**: Copy the template in`phorge/resources/sshd/phorge-ssh-hook.sh` to somewhere like`/usr/libexec/phorge-ssh-hook.sh` and edit it to have the correctsettings.Both the script itself **and** the parent directory the script resides in mustbe owned by `root`, and the script must have `755` permissions:```$ sudo chown root /path/to/somewhere/$ sudo chown root /path/to/somewhere/phorge-ssh-hook.sh$ sudo chmod 755 /path/to/somewhere/phorge-ssh-hook.sh```If you don't do this, `sshd` will refuse to execute the hook.**Create `sshd_config` for Phorge**: Copy the template in`phorge/resources/sshd/sshd_config.phorge.example` to somewhere like`/etc/ssh/sshd_config.phorge`.Open the file and edit the `AuthorizedKeysCommand`,`AuthorizedKeysCommandUser`, and `AllowUsers` settings to be correct for yoursystem.This configuration file also specifies the `Port` the service should run on.If you intend to run on a non-default port, adjust it now.**Start SSHD**: Now, start the Phorge `sshd`: sudo /path/to/sshd -f /path/to/sshd_config.phorgeIf you did everything correctly, you should be able to run this command:```$ echo {} | ssh vcs-user@phorge.yourcompany.com conduit conduit.ping --```...and get a response like this:```lang=json{"result":"phorge.yourcompany.com","error_code":null,"error_info":null}```If you get an authentication error, make sure you added your public key in{nav Settings > SSH Public Keys}. If you're having trouble, check thetroubleshooting section below.Authentication Over SSH=======================To authenticate over SSH, users should add their public keys under{nav Settings > SSH Public Keys}.Cloning a Repository====================If you've already set up a hosted repository, you can try cloning it now. Todo this, browse to the repository's main screen in Diffusion. You should seeclone commands at the top of the page.To clone the repository, just run the appropriate command.If you don't see the commands or running them doesn't work, see below for tipson troubleshooting.Troubleshooting HTTP====================Some general tips for troubleshooting problems with HTTP: - Make sure `diffusion.allow-http-auth` is enabled in your Phorge config. - Make sure HTTP serving is enabled for the repository you're trying to clone. You can find this in {nav Edit Repository > Hosting}. - Make sure you've configured a VCS password. This is separate from your main account password. You can configure this in {nav Settings > VCS Password}. - Make sure the main repository screen in Diffusion shows a clone/checkout command for HTTP. If it doesn't, something above isn't set up correctly: double-check your configuration. You should see a `svn checkout http://...`, `git clone http://...` or `hg clone http://...` command. Run that command verbatim to clone the repository.If you're using Git, using `GIT_CURL_VERBOSE` may help assess login failures.To do so, specify it on the command line before the `git clone` command, likethis: $ GIT_CURL_VERBOSE=1 git clone ...This will make `git` print out a lot more information. Particularly, the linewith the HTTP response is likely to be useful: < HTTP/1.1 403 Invalid credentials.In many cases, this can give you more information about what's wrong.Troubleshooting SSH===================Some general tips for troubleshooting problems with SSH: - Check that you've configured `diffusion.ssh-user`. - Check that you've configured `phd.user`. - Make sure SSH serving is enabled for the repository you're trying to clone. You can change this setting from a main repository screen in Diffusion by {nav Edit Repository > Edit Hosting > Host Repository on Phabricator > Save and Continue > SSH Read Only or Read/Write > Save Changes}. - Make sure you've added an SSH public key to your account. You can do this in {nav Settings > SSH Public Keys}. - Make sure the main repository screen in Diffusion shows a clone/checkout command for SSH. If it doesn't, something above isn't set up correctly. You should see an `svn checkout svn+ssh://...`, `git clone ssh://...` or `hg clone ssh://...` command. Run that command verbatim to clone the repository. - Check your `phorge-ssh-hook.sh` file for proper settings. - Check your `sshd_config.phorge` file for proper settings.To troubleshoot SSH setup: connect to the server with `ssh`, without running acommand. You may need to use the `-T` flag, and will need to use `-p` if youare running on a nonstandard port. You should see a message like this one: $ ssh -T -p 2222 vcs-user@phorge.yourcompany.com phorge-ssh-exec: Welcome to Phorge. You are logged in as alincoln. You haven't specified a command to run. This means you're requesting an interactive shell, but Phorge does not provide an interactive shell over SSH. Usually, you should run a command like `git clone` or `hg push` rather than connecting directly with SSH. Supported commands are: conduit, git-receive-pack, git-upload-pack, hg, svnserve.If you see this message, all your SSH stuff is configured correctly. **If youget a login shell instead, you've missed some major setup step: review thedocumentation above.** If you get some other sort of error, double check thesesettings: - You're connecting as the `vcs-user`. - The `vcs-user` has `NP` in `/etc/shadow`. - The `vcs-user` has `/bin/sh` or some other valid shell in `/etc/passwd`. - Your SSH private key is correct, and you've added the corresponding public key to Phorge in the Settings panel.If you can get this far, but can't execute VCS commands like `git clone`, thereis probably an issue with your `sudoers` configuration. Check: - Your `sudoers` file is set up as instructed above. - You've commented out `Defaults requiretty` in `sudoers`. - You don't have multiple copies of the VCS binaries (like `git-upload-pack`) on your system. You may have granted sudo access to one, while the VCS user is trying to run a different one. - You've configured `phd.user`. - The `phd.user` has read and write access to the repositories.It may also be helpful to run `sshd` in debug mode: $ /path/to/sshd -d -d -d -f /path/to/sshd_config.phorgeThis will run it in the foreground and emit a large amount of debugginginformation when you connect to it.Finally, you can usually test that `sudoers` is configured correctly bydoing something like this: $ su vcs-user $ sudo -E -n -u daemon-user -- /path/to/some/vcs-binary --helpThat will try to run the binary via `sudo` in a manner similar to the way thatPhorge will run it. This can give you better error messages about issueswith `sudoers` configuration.Miscellaneous Troubleshooting============================= - If you're getting an error about `svnlook` not being found, add the path where `svnlook` is located to the Phorge configuration `environment.append-paths` (even if it already appears in PATH). This issue is caused by SVN wiping the environment (including PATH) when invoking commit hooks.Moving the sshd Port====================If you want to move the standard (administrative) `sshd` to a different port tomake Phorge repository URIs cleaner, this section has some tips.This is optional, and it is normally easier to do this by putting a loadbalancer in front of Phorge and having it accept TCP traffic on port 22and forward it to some other port.When moving `sshd`, be careful when editing the configuration. If you get itwrong, you may lock yourself out of the machine. Restarting `sshd` generallywill not interrupt existing connections, but you should exercise caution. Twostrategies you can use to mitigate this risk are: smoke-test configuration bystarting a second `sshd`; and use a `screen` session which automaticallyrepairs configuration unless stopped.To smoke-test a configuration, just start another `sshd` using the `-f` flag: sudo /path/to/sshd -f /path/to/config_file.editedYou can then connect and make sure the edited config file is valid beforereplacing your primary configuration file.To automatically repair configuration, start a `screen` session with a commandlike this in it: sleep 60 ; mv sshd_config.good sshd_config ; /etc/init.d/sshd restartThe specific command may vary for your system, but the general idea is to havethe machine automatically restore configuration after some period of time ifyou don't stop it. If you lock yourself out, this can fix things automatically.Now that you're ready to edit your configuration, open up your `sshd` config(often `/etc/ssh/sshd_config`) and change the `Port` setting to some other port,like `222` (you can choose any port other than 22). Port 222Very carefully, restart `sshd`. Verify that you can connect on the new port: ssh -p 222 ...Now you can move the Phorge `sshd` to port 22, then adjust the valuefor `diffusion.ssh-port` in your Phorge configuration.You can set up and enable this systemd unit to start the second sshddaemon on every reboot:```name=/etc/systemd/system/phorge-ssh.service,lang=ini[Unit]Description=Phorge sshdDocumentation=https://we.phorge.it/book/phorge/article/diffusion_hosting/#sshd-setupAfter=network.target auditd.service[Service]ExecStartPre=/usr/sbin/sshd -t -f /path/to/config_file.editedExecStart=/usr/sbin/sshd -f /path/to/config_file.editedExecReload=/usr/sbin/sshd -t -f /path/to/config_file.editedExecReload=/bin/kill -HUP $MAINPIDKillMode=processRestart=on-failureRestartPreventExitStatus=255Type=notifyRuntimeDirectory=sshdRuntimeDirectoryMode=0755[Install]WantedBy=multi-user.targetAlias=phorge-sshd.service```No Direct Pushes================You may get an error about "No Direct Pushes" when trying to push. This meansyou are pushing directly to the repository instead of pushing throughPhorge. This is not supported: writes to hosted repositories must gothrough Phorge so it can perform authentication, enforce permissions,write logs, proxy requests, apply rewriting, etc.One way to do a direct push by mistake is to use a `file:///` URI to interactwith the repository from the same machine. This is not supported. Instead, useone of the repository URIs provided in the web interface, even if you'reworking on the same machine.Another way to do a direct push is to misconfigure SSH (or not configure it atall) so that none of the logic described above runs and you just connectnormally as a system user. In this case, the `ssh` test described above willfail (you'll get a command prompt when you connect, instead of the message youare supposed to get, as described above).If you encounter this error: make sure you're using a remote URI given toyou by Diffusion in the web interface, then run through the troubleshootingsteps above carefully.Sometimes users encounter this problem because they skip this whole documentassuming they don't need to configure anything. This will not work, and youMUST configure things as described above for hosted repositories to work.The technical reason this error occurs is that the `PHABRICATOR_USER` variableis not defined in the environment when commit hooks run. This variable is setby Phorge when a request passes through the authentication layer that thisdocument provides instructions for configuring. Its absence indicates that therequest did not pass through Phorge.Next Steps==========Once hosted repositories are set up: - learn about commit hooks with @{article:Diffusion User Guide: Commit Hooks}.