• BuildFarm

Build Farm

We use Buildbot for automated build and test.

Bitten

For the developer

During the automated build, buildbot runs rosautomake on each package in the tree. rosautomake is just like rosmake, except that it will skip any package that contains a file called ROS_BUILD_BLACKLIST. This file is used to mark packages that are known to be broken, and therefore should be excluded from the build.

The goal is for the automated build to always succeed; it should fail only when a bug is introduced. Blacklisted packages are excluded from the build in order to allow the build to succeed without them, and only report failure (via web interface and email) when a package that's expected to build becomes broken.

The ROS_BUILD_BLACKLIST file is usually added by the buildbot admin, after it is seen that the package repeatedly fails to build, and the package maintainer doesn't fix the bug. If you find this file in your package, it's because your package is broken. Shame on you! But there's hope! Look in the ROS_BUILD_BLACKLIST file to find links to the output from a broken build, plus potentially links to relevant Trac tickets.

If you believe that you've fixed your broken package, then svn delete and svn commit the ROS_BUILD_BLACKLIST file. Your package will be built in the next run.

For the administrator

Buildbot is a distributed build and test system. For each project being build, one buildmaster runs, on some machine. For each project, any number of buildslaves may run. The buildmaster gets all the configuration as to what and how to build.

We currently run 2 builds:

• ros
• personalrobots

Current configuration at Willow:

• buildmaster runs on pr.willowgarage.com

  • runs as www-data, config is in /var/www/pr.willowgarage.com/buildbot/ros and /var/www/pr.willowgarage.com/buildbot/personalrobots

  • buildmaster(s) restart on reboot, via cron

  • To restart a master, e.g., for ros, become www-data on pub1 and do:

    • buildbot restart /var/www/pr.willowgarage.com/buildbot/ros
      

• buildslaves run in various places, including wgs7 and wgs8.
  • pattern is that the build runs as rosbuild, which has a home directory of /u/rosbuild, but the build happens in /home/rosbuild/slaves, which is local to each buildslave (keeps the heavy disk access local).

  • buildslave(s) restart on reboot, via cron

  • To restart a slave, e.g., for ros on wgs7, become rosbuild on wgs7 and do:

    • buildbot restart /home/rosbuild/slaves/ros
      

Adding a buildslave

Currently, these are Willow-specific instructions.

Here's what to do on the slave machine, which is assumed to be in the NIS network, with /wg/stor1 mounted as /u via NFS. There will be a rosbuild user, with a home directory /u/rosbuild. This home directory contains cached SSL certificates for all https servers that are visited during the build (e.g., for SVN checkouts), and the .bashrc sets up ccache to use /home/rosbuild/.ccache.

(as root) Setup buildbot:

1. Install buildbot and ccache. On Ubuntu:

   apt-get install buildbot ccache
   

2. Create a local directory to build in:

   mkdir /home/rosbuild
   chown rosbuild.wgusers /home/rosbuild
   

(as rosbuild) Setup the build slave environment:

1. Create a place to put the slave builds:

   cd /home/rosbuild
   mkdir slaves
   mkdir slaves/ros
   mkdir slaves/personalrobots
   

2. Initialize the slave directories (to get the values referenced in angle brackets below, ask Brian):

   buildbot create-slave /home/rosbuild/slaves/ros <host:port> <slave-name> <slave-pw>
   buildbot create-slave /home/rosbuild/slaves/personalrobots <host:port> <slave-name> <slave-pw>
   

3. Edit the host and info files:

   echo "Your Name <you@example.com>" > /home/rosbuild/slaves/ros/info/admin
   echo `uname -a` `cat /etc/issue` > /home/rosbuild/slaves/ros/info/host
   echo "Your Name <you@example.com>" > /home/rosbuild/slaves/ros/info/admin
   echo `uname -a` `cat /etc/issue` > /home/rosbuild/slaves/personalrobots/info/host
   

(as root) Install ROS prerequisites, which are given in ROS/Installation.

(as root) Install prerequisites for packages in the personalrobots repository, which are stored as <sysdepend> tags in the package manifests:

1. There's currently no automated way to do this. For now, you can do the following on another machine, with a working ROS installation:

   roscd rospack
   ./rosallsysdeps  | grep ubuntu | awk {'print $6'} | sort -u
   

   You'll get something like this:

   autoconf
   automake
   freeglut3-dev
   jam
   libcurl3-openssl-dev
   libcurl4-openssl-dev
   libdc1394-13-dev
   libfltk1.1-dev
   libgl1-mesa-dev
   libglu1-mesa-dev
   libgtk2.0-dev
   libltdl3-dev
   libqt3-mt-dev
   libtool
   libwxgtk2.8-dev
   libxaw7-dev
   libxext-dev
   libxml2-dev
   libxt-dev
   libxxf86vm-dev
   libzzip-dev
   pkg-config
   python-sip4-dev
   python-wxgtk2.8
   scons
   sip4
   

   You can almost just pass this list into sudo apt-get. The one complication is that this list contains the union of packages for all Ubuntu versions, which might not be what you want. You can further filter on version, e.g.:

   ./rosallsysdeps  | grep ubuntu | grep hardy | awk {'print $6'} | sort -u
   

   But because a given package may specify its <sysdepends> for one version and not another, you may not get a complete list. A system is underway for making the process work better.

(as rosbuild) Start the slaves:

1. Run buildbot:

   buildbot start /home/rosbuild/slaves/ros
   buildbot start /home/rosbuild/slaves/personalrobots
   

2. Configure buildbot to start on boot, by adding these lines to rosbuild's crontab (edit the crontab by running crontab -e):

   @reboot /usr/bin/buildbot start /home/rosbuild/slaves/ros
   @reboot /usr/bin/buildbot start /home/rosbuild/slaves/personalrobots