Phabricator Administrator's Handbook/Continuous integration

Screenshot some "Passed" and "Failed" inside a Phabricator history page.

Phabricator can be configured to support some continuous integration (CI) workflows, thanks to a component called Harbormaster.

Start reading the official Harbormaster User Guide to understand some important generic concepts.

What do you need

• 1/2 hours
• knowledge about the Phabricator's Diffusion component (the one handling your repositories)

Installation

Harbormaster is already part of Phabricator but it's not showed as default. It's a prototype application.

Check how to enable prototype applications:

https://secure.phabricator.com/book/phabricator/article/prototypes/

Preamble

Often people starting with in-house continuous integration just want to react on commits and run a command, but this is extremely reductive and could become dangerous.

That's why new Phabricator users are often surprised to learn that Phabricator does not allow this kind of "super-simple" solutions. This is good because, before being able to make big mistakes, we will need to read some documentation.

Security concerns

First of all, do not try to run a direct command on your working repository. Never. This could break that repository and, moreover, you could also end in compromising your whole Phabricator infrastructure.

Spend some time in implementing a standard and clean solution to keep your repositories safe and untouched by scripts, and without running these scripts with the same Phabricator privileges. For example, running scripts on other machines, on other working copies, and anyway with low-privileged ad-hoc users, not Phabricator-related.

Read the Drydock Securty Official Documentation for further notes.

Setup solutions

To be very generic, you have two choices:

1. configure Phabricator to connect to some SSH servers and do everything from there (clone, build, ecc.)
2. delegate the build to other services

The first solution is interesting because allows to use your infrastructure. This can be configured using a Phabricator component called Drydock.

The second solution works just calling an external service's API. It's interesting if you already have a build system, let's say for example Jenkins.

Setup 1: on own hardware (using Drydock)

A continuous integration flow in Phabricator with Drydock.

You can configure Phabricator's Harbormaster component to build stuff using your own hardware, thank to a component called Drydock. Drydock allocates your resources.

Here the complete flow of a working real-world example:

1. a developer run git push to publish local edits in Phabricator (or use another source version control tool supported in Phabricator, like Subversion or Mercurial)
2. Phabricator checks if that commit triggers a Rule, and eventually do an action. The Herald component defines these rules.
3. an Herald rule is matched, and the action is to run a "Build Plan" configured in Harbormaster. It starts its Steps.
4. the first Step of the Build Plan could be to ask a Working Copy (we say, we "lease" a Working Copy - because we do not work directly on the same repository used by Diffusion and we just get or create another copy)
5. the Drydock component gives that working copy, checking if there is an available host with a ready working copy. It knows your configured infrastructure thanks to your configured Almanac inventary.
6. the next step of the Build Plan can be "run a command" on that working copy (technically it runs a command on a "Drydock lease")
7. if the command quits successfully, you will see a Passed mark near your commit or Failed otherwise.

Let's start implementing this.

We first suggest to read this example configuration from the Phabricator Discussion forum.

Describe your Networks and Devices

First of all you need to describe some Devices inside your Almanac inventory.

What we mean with a "Device":

• a physical server accessible with SSH
• a virtual server (a VPS, a docker run, a virtual machine, etc.) accessible with SSH

You know what kind of device you want to dedicate to your build system. Maybe it's a virtual server, maybe is exactly your same server (but with a low-privileged User).

For now, let's just describe it inside your Almanac inventory:

1. Define an Almanac Network: You must have at least one Network. Usual names are "public" or "private" or "local network". Your servers will be associated to this.
2. Define a Almanac Device: Give a name to the first server you want to dedicate for your build system. This server should be accessible from SSH from Phabricator, so click on Add Interface, pick the previous Network, insert its IP address / domain, and type the standard SSH port 22.
3. Define a Almanac Service: Open Almanac and define a Service selecting Drydock: Resource Pool as Service Type. This will be a container of available servers so an name may be "Buildservers Pool". Click on "Add Binding" and select the previous Device.

Prepare your build server

At this point you should know exactly which server you want to dedicate for your build system. You should know how to connect to SSH to that server (if no, read the SSH book).

Make sure you can connect to your server using SSH with public keys (tip: create an additional SSH key for this purpose, because you will need to publish the related private key in Phabricator to allows it to connect to your server):

ssh foobot@server_build_01

Once you are inside your server, make sure that you are allowed to clone your repositories. Try to clone one of your repositories. If your repositories are public you are OK. Instead, if your repositories are under some visibility restrictions you should create a dedicated Phabricator Bot (a Bot is an User of type "Bot") and put that user in the right projects. You can create a Bot from the Phabricator Users page. The bot can be called for example "Foo Bot @ server_build_01") and you have to add it in the right projects in order to allow that user to read your repositories.

You may want to generate an SSH key from the server and publish its public key in its User Preferences (the user preferences of the Bot you just created). In short, make sure you can clone your repositories from your server from the user you choosed (foobot), using standard Phabricator authorizations.

Tip: actually the user you choosed (foobot) should be able to write inside the /var/drydock directory in order to clone working copies.

Read more from the Drydock hosts Documentation.

Setup Drydock Blueprints

Drydock menu with Blueprints and Resources

Now you should define a couple of "Blueprints". A Blueprint is a "goal" to be reached by the Drydock component. Give a read the Drydock Blueprint Documentation first.

In short you need to configure two Drydock Blueprints:

1. Create a Blueprint of type Almanac Hosts - this Blueprint will choose wich Hosts are available, so an example name can be "Host Leaser". In the field Almanac Services select the one you created (maybe called "Buildservers Pool"). Remember to set up a valid SSH public key to access the host, as described in #Prepare your build server.
2. Create a Blueprint of type Working Copy - this Blueprint will provide Working Copies so an example name can be "Working Copy Leaser from Buildservers Pool". In the field Use Blueprint select the previously created Blueprint (maybe called "Host Leaser").

Now run this command from the Phabricator's root:

./bin/drydock lease --type host

You should see a similar output:

$ ./bin/drydock lease --type host
Queued lease for activation:

        https://example.com/drydock/lease/1/

Waiting for daemons to activate lease...
<Lease Queued> Lease queued for acquisition.
<Lease Acquired> Lease acquired.
<Lease Activated> Lease activated.
Activation complete. This lease is permanent until manually released with:

        $ ./bin/drydock release-lease --id 1

Lease activated in 2,277ms.

You can further read the Drydock Working Copies Official Documentation for further notes.

Setup Harbormaster Build Plan

Now go in the Harbormaster page and click on "Create Build Plan".

Add two Build Step:

1. Create a Build Step of type Lease Working Copy. In the field Use Blueprint pick the previously created Drydock Working Copy Blueprint (maybe called "Working Copy Leaser from Buildservers Pool"). An example name can be "Lease Working Copy from from Buildservers Pool". The Artifact Name is what this step produces, so an example name can be "Leased Working Copy from from Buildservers Pool".
2. Create a Build Step of type Drydock: Run Command. In the field Drydock Lease put the same value writed in the Artifact name field of the previous Step (maybe called "Leased Working Copy from from Buildservers Pool"). In the Command field type the command that will be run inside your repository.

Tip: the working directory of the command will be the pathname to a working copy. You should not try to predict its complete pathname because it's generated inside.

Tip: the working copies will be cloned down in /var/drydock

Tip: the command will be executed with the privileges of the SSH user you prepared in #Prepare your build server

Run the Build Plan after an event

An example of an Herald rule triggering a Build Plan.

Now you may want to configure Phabricator to automatically run a Build Plan after an events. For example:

• run a Build Plan after a trusted user commits in a repository - see #Setup Herald to react on commits
• run a Build Plan after an untrusted user sends or modifies a patch for review - see #Setup Herald to react on Differential revisions

Setup Herald to react on commits

You may want to automatically run a Build Plan when someone do a commit on a repository. In this case:

1. open the homepage of the Herald component and click on Create Herald Rule
2. select New Rule For: Commits
3. select Rule Type: Object - in order to attach your Herald rule to a specific repository
4. type your repository code - note that it starts with an uppercase R, something like R123
5. in the Conditions field you may want to select Always
6. in the Action field you may want to add Run Build Plans and select your Build Plan.

Setup Herald to react on Differential revisions

Example of a Diffusion repository with an external Staging Area configured.

You may want to automatically run a Build Plan when someone proposes a patch, from Differential.

In Phabricator an user can propose a patch (a Differential revision) for a repository. The user can submit this patch using a command line tool called Arcanist (with the command arc diff). Then you may want to check if this patch will breaks everything once merged (landed) in your repository.

The problem with this approach is that you can't just run a Build Plan in this case. In fact, at this time you have not a repository with that patch committed.

This is why this solution is similar to the #Setup Herald to react on commits but we have to introduce a Staging Area.

Here the steps:

1. create a new repository (it can be in Diffusion or not) and make sure that a generic user is allowed to push in it. This will be your Staging Area.
2. now turn back to your original repository and open its Diffusion page. Editing it, you should notice a Staging Area menu. From that page, paste the URL of your new repository
3. open the homepage of the Herald component and click on Create Herald Rule
4. select New Rule For: Differential Revisions
5. select Rule Type: Global
6. in the Conditions field you may want to select Repository Projects include any of and then select your projects
7. in the Action field you may want to add Run Build Plans and select your Build Plan.

See also the Official Harbormaster Documentation, in the Change Handoff section about Manual Staging Area.

Setup 2: using an external service

The Jenkins Dashboard, an example of external service.

Another common solution is adopting an external build service out of Phabricator. This solution can be better if you already have experience with an external build service.

Note that, when calling an external service, Phabricator may be not able to wait for the output (because Phabricator makes an HTTP request to your external service, but obviously the service does not end the build at the end of that short request). So, you may need to also configure another little daemon to interrogate your external service, wait for your build, and eventually notify Phabricator about that build status.

Some not-exhaustive list of resources:

• Jenkins and Phabricator integration

Troubleshooting

Here a not-exhaustive list of well-known errors you can see after implementing the #Setup 1: on own hardware using Drydock.

Troubleshooting Lease PHID-... never activated

If you see the error Lease PHID-... never activated or a similar stack trace:

PhabricatorWorkerPermanentFailureException: Lease "PHID-..." never activated. in .../phabricator/src/applications/harbormaster/step/HarbormasterLeaseWorkingCopyBuildStepImplementation.php:91
Stack trace:
#0 .../phabricator/src/applications/harbormaster/worker/HarbormasterTargetWorker.php(70): HarbormasterLeaseWorkingCopyBuildStepImplementation->execute(Object(HarbormasterBuild), Object(HarbormasterBuildTarget))
#1 .../phabricator/src/infrastructure/daemon/workers/PhabricatorWorker.php(124): HarbormasterTargetWorker->doWork()
#2 .../phabricator/src/infrastructure/daemon/workers/storage/PhabricatorWorkerActiveTask.php(158): PhabricatorWorker->executeTask()
#3 .../phabricator/src/infrastructure/daemon/workers/PhabricatorTaskmasterDaemon.php(22): PhabricatorWorkerActiveTask->executeTask()
#4 .../libphutil/src/daemon/PhutilDaemon.php(219): PhabricatorTaskmasterDaemon->run()
#5 .../libphutil/scripts/daemon/exec/exec_daemon.php(131): PhutilDaemon->execute()
#6 {main}

Check again #Setup Drydock Blueprints and see the logs of your Blueprints. Maybe you have not activated any host.

Troubleshooting Resource activation failed: [CommandException] Command failed with error #128!

If you see this error or a similar one:

Resource activation failed: [CommandException] Command failed with error #128!
COMMAND
ssh '-o' 'LogLevel=ERROR' '-o' 'StrictHostKeyChecking=no' '-o' 'UserKnownHostsFile=/dev/null' '-o' 'BatchMode=yes' -l ...

STDOUT
(empty)

STDERR
LOT OF ERRORS HERE

Look carefully at the STDERR part:

Troubleshooting Could not create directory .ssh

• If you have a:

Could not create directory '.../.ssh'

It means the Phabricator user daemon is trying to connect to your build server using SSH so it needs a writable $HOME/.ssh directory to write the $HOME/.ssh/known_hosts file etc. Maybe the home of the Phabricator user daemon is not writable by him.

If you do not know what is your Phabricator user daemon, is the one running your daemons. You set up it during your installation.

Troubleshooting Could not create leading directories of /var/drydock

If you have a:

could not create leading directories of '/var/drydock/...': Permission denied

The user building stuff can't write in /var/drydock or that pathname does not exist. Try to read again the section #Prepare your build server. Note that actually that pathname can't be customized.

Troubleshooting fatal: Could not read from remote repository

If you have a:

fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists.

Go back to section #Prepare your build server. Login inside your build server and try to clone repositories from it. You may have to authorize the user of the server with a dedicated Bot account in Phabricator, configuring an SSH key.