Creating a Nightly Build

The Nightly Build

Prerequisites

SSH Account for the AROS Project on sourceforge.net
Space to store all the required source for your chosen packages,
Space to store the generated binaries and log files.
A valid svn account for https://svn.aros.org/ with read-access.
An email account subscribed to the AROS dev mailing list.

Terms Used

<SCRIPTDIR>: The location the nightly scripts are stored in
<Sourceforge SSH Account Name>: A valid SSH account on sourceforge with access to AROS.
<Nightly-Build User Name>: The local user account you will build binaries with.
<Dev Mailing List eMail Address>: An email account with access to the AROS Dev mailing list.; This will be used to send the log to the mailing list.
<Your Servers Config-Name>

Nightly Package Names

The build script uses package names to identify what to build for a given server. the packages tell the build script which files in <SRCDIR>/pkg/ to run, which in turn creates the chosen binaries.

Valid Package Names

Files which are common to all platform generally have a name that is just their target, with all as the build target. Currently defined ones are

Common Package Names

  sources.svn.all
  changelogs.svn.all
  documentation.svn.all

Arch Specific Package Names

  sdk.svn.pc-i386
  sdk.svn.pc-x86_64
  
  contrib.svn.pc-i386
  contrib.svn.pc-x86_64
  
  bootiso-grub.svn.pc-i386
  bootiso-grub2.svn.pc-i386
  bootiso-grub2.svn.pc-x86_64
  
  bootfloppy.svn.pc-i386
  
  hosted.svn.linux-i386
  hosted.svn.linux-x86_64
  hosted.svn.linux-ppc

Disallowed Names

  bootiso                            generic routine used by the arch specific nightly targets
  hosted
  sdk

Prepairing the system

Firstly you will want to set up a user account on your system that will be used to perform the build. create a user

  adduser <Nightly-Build User Name>

The first step is to obtain the nightly build scripts. cd to the location you will perform the build in and checkout the scripts as follows -:

  cd /home/<Nightly-Build User Name>
  mkdir -p MyNightlyBuildDir
  cd MyNightlyBuildDir
  svn checkout https://svn.aros.org/svn/aros/trunk/AROS/scripts/nightly

you will have a new subdirectory called nightly that contains all the scripts used to build the nightly binaries. from here on in we will refer to this directory as <SCRIPTDIR>.

now cd into that directory and make a directory tree to hold the AROS source

  cd <SCRIPTDIR>
  mkdir -p Source
  mkdir -p Source/Contrib
  mkdir -p Source/Documentation
  mkdir -p Source/Necessary

Finaly create an initial cache of the svn files used to build AROS

IMPORTANT: You must set the correct LOCALE for your system at this point!!

type the following -:

  cd Source
  export LANG=en_US.ISO8859-1

then checkout all the source repositories needed to perform your nightly build packages.

ie...

  svn checkout https://svn.aros.org/svn/aros/trunk/AROS
  cd Contrib
  svn checkout https://svn.aros.org/svn/aros/trunk/contrib
  cd ../Necessary
  svn checkout https://svn.aros.org/svn/aros/trunk/contrib/necessary

You may omit complete contrib if you don't want to bundle it within your archive. Just leave the corresponding directory empty.

Prepairing your servers config file

Create a plain textfile in the cfg directory called <Your Servers Config-Name> Open it in a text editor and add the packages you'd like to build.

For example ..

  CFG_PACKAGES=(sources.svn.all documentation.svn.all sdk.svn.pc-x86_64 bootiso-grub2.svn.pc-x86_64)

To enable sending an email report of the build results to the dev mailing list add the following lines ..

  CFG_MAIL_ENABLE=yes
  CFG_MAIL_CC=<Dev Mailing List eMail Address>

And to enable uploading the binaries so they may be listed on AROS.org add ..

  CFG_UPLOAD_ENABLE=yes
  CFG_DS_USER=<Sourceforge SSH Account Name>
  CFG_BACKUP_ENABLE=no

Performing your first run

The first time you check the system is operational ... perform a manual dry run by doing the following

  cd <SCRIPTDIR>; ./build <Your Servers Config-Name>

And watch as it displays all your build output.

If everything went well and you have enabled the email logging in your server config, you should see a confirmation email on the AROS developer mailing list confirming the build completed successfully. If you have also enabled uploading the files, and have a valid AROS SSH account on sourceforge, the binaries should be uploaded into the temporary storage location ready to be moved into the live folders on the AROS.org download pages

Setting it up to run automatically

You will need to create a cron job for the build step we performed doing the dry run. You will want to add a command that is performed once every 24hrs, which runs the build script with your server's configuration and uploads the created archives to AROS.org.

   0 2 * * * ( cd <SCRIPTDIR> && ./build <Your Server's Config-Name> ) > /dev/null

This will make the build machine attempt to build and upload AROS at 2am. All output will be directed to NULL (except of course the generated logfiles)

Adding the binaries to the AROS.org site

If you are simply providing binaries for existing AROS targets, the previous steps should be enough to add the files to the AROS.org site. If however you wish to provide new flavours of AROS then you will need to update the AROS.org site with information about them. Read on...

The scripts and page definitions to list the binaries on AROS.org are maintained in svn at https://svn.aros.org/svn/aros/trunk/documentation

To edit the files you'll need to checkout the repository and customise to suit.

listing new downloads

If you are providing binaries for AROS targets that don't currently exist, you will need to update the files which list them on AROS.org.

Edit the db/download-descriptions/<Language> files to list descriptions of the packages you build. make sure and follow the naming convention used by the existing packages.

Troubleshooting

Corrupt localised filenames in the binaries

This usually means the source files used have utf8 filenames in the source repository caches resulting from the initial checkout being done when the system locale was set to utf8.

to fix it change directory into each source directory in turn and delete all files and directories EXCEPT the top level .svn directory.

now type ..

  export LANG=en_US.ISO8859-1

.. and finally do an "svn update" in each directory.

Maintaining the Main Nightly Build Machine

The main nightly build machine has the additional responsibility of running scripts to move the uploaded binaries to the live AROS.org download pages and clean up the upload directory. Other build machines don't need to run these scripts.

Step1

Moving the uploaded binaries to the live AROS.org download pages.

You will need to create a cron job that runs at regular intervals during the day, currently every hour. This job however isn't performed locally, and will require an AROS SSH account on sourceforge. Its goal is to move the archives uploaded by the various build machines into the correct folder on the AROS.org site so that they become available for download. It also deletes any old versions of these files so that only the most up-to-date version is available (there isn't enough space to keep multiple versions of each file).

   00 * * * * ssh <Sourceforge SSH Account Name> 'bash -l -c aros/scripts/nightly-update' > /dev/null

This will make the server transfer any files present in the upload dirextory to the AROS.org site once an hour. Again all output is sent to NULL.

Step2

This step removes incomplete or broken nightly build files from the upload directory.

   51 7 * * * ssh <Sourceforge SSH Account Name> 'bash -l -c aros/scripts/nightly-clean' > /dev/null

This will clean up the uploads directory on the AROS.org server at 7:51am each day.