em_autobuild

Langue: en

Version: Sun 11 Jan 2009 19:55:45 GMT (debian - 07/07/09)

Section: 1 (Commandes utilisateur)

NAME

em_autobuild - emdebian target package autobuilder

SYNOPSIS

em_autobuild [[-a] | [--arch] ARCHITECTURE] [[-e] | [--errexit]] [[--script] FILENAME] [[-p] | [--pbuilder]] [[--package] PACKAGE] [[-l] | [--log]] [[-u] | [--handler]] [[--machine-path]PATH] [[--machine] NAME [[--variant] NAME]] [[--simulate]] [[--]OPTIONS] [--skip-compare]
em_autobuild [[-?] | [-h] | [--help] | [--version]]

DESCRIPTION

Parses the /usr/share/emdebian-tools/emdebian.crossd suite script (by default) or a user-specified suite script, collates a list of base and required binary packages, converts each to the respective source package using the apt-cross cache and attempts to build each source package with emsource --build-dep -b $package.

The same machine:variant support is available as in emsource itself except when using --log because emsource is then run inside the chroot instead of in the development environment.

The build sequence halts when the first package fails to build cleanly - this includes lintian errors generated by the Emdebian lintian check set. (Errors from the standard Debian checks are ignored.)

The logfile support is intended for unattended autobuilds via at or cron.

Whilst em_autobuild can be run as a normal user, various factors can complicate things when using a logfile because all output is redirected to the logfile. This can cause the autobuild to just halt without the user being able to provide the input necessary to give the sudo passphrase. Similarly, users who have an SVN username at www.emdebian.org will find that the SSH passphrase cache can time out, again without the ability to supply the passphrase. The easiest option is not to use a log file.

One complication of this is that machine:variant support is not implemented if --log is used.

To use --log on a development system, run em_autobuild as the root user and in the empdebuild cross-building chroot. To use your existing empdebuild chroot:

1. Ensure your empdebuild chroot is up to date:
 $ sudo empdebuild update
         
 
2. Use the --pbuilder so that em_autobuild can run emsource within the cross-building chroot. (The patches do need to be up to date to use this method.)
3. Echo the command to at for the root user:
 $ sudo echo "em_autobuild -l -o -a arm -p" | at now
         
 

This way, sudo is only needed once, the chroot is unpacked as root and emsource and emdebuild run within the chroot (your SVN username is not copied into the chroot).

COMMANDS

Options for dpkg-buildpackage can be given after the -- option. The default options are -uc -us which cannot be overridden. Also certain commands do not make sense in this context, e.g. -a for architecture will be silently dropped.

-o|--omit-current

Exclude packages that appear to be up to date. When running em_autobuild via at or cron, remember that running scripts via the crontab for root means that Emdebian::Tools looks for data in /root/ instead of $HOME.
The list of packages to omit is obtained via emtargetcmp.
Another change that can catch out cron usage is incomplete locale configuration (causing lots of unnecessary perl warnings from the perl components of Emdebian::Tools).
An example script to run an autobuilder:
 #!/bin/sh
 
 export LANG=C
 export LC_ALL=C
 export LANGUAGE=C
 export ANSI_COLORS_DISABLED=true
 export HOME=/home/neil/
 emtargetcmp
 em_autobuild --pbuilder --omit-current --log
           
 

--package PACKAGE

Auto-build just a single package - usually used with --log.

--simulate

Simulate an autobuild by processing the list of packages and calculating the full set of dependencies that would be built by em_autobuild.

-h|--help

print the usage message and exit.

--version

print the usage message and exit.

OPTIONS

-a|--arch ARCHITECTURE

Override the dpkg-cross default architecture for this operation on the chroot.

-e|--errexit

Allow the autobuild to halt at the first build that fails. Default behaviour is to carry on and build whatever packages can be built, recording those that fail using embug.

-s|--script FILENAME

Override the default package selection and installation script with a customised debootstrap suite script (written in shell and compatible with whichever shell interpreter is to be installed on the target).
Some customised scripts are provided with emdebian-tools. The default uses the standard Emdebian 'busybox' package with 'dpkg' and 'apt'. Replacement scripts need to be full debootstrap suite shell scripts that specify how to complete the first and second stage installations.
Customised scripts packages with emdebian-tools include scripts for a root filesystem including libgtk2.0-0 and a complete GPE root filesystem.

-u|--handler

Provide an executable script to do the upload of the build log and/or packages.
To upload build logs after unattended autobuilds, specify a script which will be called once for each package with the full path to the build log file as the only argument. The script can then use whatever methods are supported in your environment to upload the build log. --handler is ignored if --log is not used.
If the handler script also uploads the packages (e.g. using emrecent in noninteractive mode), the handler script needs to check that the build was successful then upload the packages before appending the .upload file to the end of the build log for that package. This allows the buildd.php report script to list the package as "Installed" instead of "pending".

-p|--pbuilder

Run the autobuild in an empbuilder chroot (which must already exist and needs to be updated separately).

-l|--log


emdebuild logs only the actual build process. em_autobuild --log wraps that build log with details of the dependency installation (native and cross) at the start and the .changes contents at the end (if successful).
When using --log, the emsource build log is written out to the package directory beneath $WORK/buildd, e.g. /opt/emdebian/buildd/a/apt/trunk/apt-arm-1213105765.log. Log file support is intended for autobuilders where the detailed build log output is not useful when emailed to the admin by at or cron, instead the logs are intended for a website or similar output. Therefore, in --log mode, em_autobuild simply outputs the list of packages being processed. The $WORK/buildd directory could then be made available over HTTP etc.
When using em_autobuild, note that enabling logging can cause you to be unable to enter passwords for SSH keys etc. (This is expected as --log is intended for use on unattended buildd machines that do not have SVN commit access and where the autobuild is likely to happen inside a chroot so that sudo is not an issue). When testing, consider using -p as well as -l or extending the length of time that your SSH and/or sudo passwords are cached.
To monitor a logged autobuild (which is normally silent), use htop (use the right cursor arrow key to see the full command listing and F5 to get a tree listing).

--machine-path PATH

Override the default path to machine and variant configuration. By default, emsandbox uses ${WORK}/machine where $WORK is the working directory specified to emdebian-tools in the debconf configuration. The specified path must already exist and contain the relevant packages.conf configuration as well as the setup.sh and config.sh shell scripts (which may be empty).

-m|--machine MACHINE

Load machine specific configuration data from your Emdebian working directory. If no variant is specified, config is read from $WORK/machine/$MACHINE/default/ where $WORK is the work directory specified in debconf for emdebian-tools.

-v|--variant VARIANT

Load variant specific configuration data from your Emdebian working directory. Requires --machine. Configuration data is read from $WORK/machine/$MACHINE/$VARIANT/ where $WORK is the work directory specified in debconf for emdebian-tools.

--skip-compare

em_autobuild normally runs emtargetcmp to check progress against the Debian packages and identify which packages need to be built to keep in step with Debian. If you do not need this functionality or if the memory requirements of this comparison become a problem, use this function to skip the check. Either identify the packages that are falling behind using other methods or run emtargetcmp on a faster machine and pass the list to em_autobuild via a loop, using the --package option.

MACHINE VARIANTS

emsandbox supports a set of customisation routines for each combination of machine and variant, allowing the rootfs to be customised to specific variants of a specific machine. Configuration data is stored in the machine subdirectory of your Emdebian work directory. Using the -m option to emsandbox loads packages.conf from the $WORK/machine/$MACHINE/default subdirectory prior to starting debootstrap. Once the first stage install is complete, emsandbox calls setup.sh from the same directory, passing the location and architecture of the tarball, so that other fine tuning can take place prior to creating the tarball. At this stage, any operations inside the rootfs must not try to execute any binaries within the rootfs. Immediately before creating the tarball, config.sh is copied into the /machine/$MACHINE/default/ directory of the rootfs, ready to be called when emsecondstage has completed the second stage of the debootstrap process.

Skeleton versions of packages.conf, setup.sh and config.sh are available in /usr/share/emdebian-tools/machine/.

packages.conf is intended to be the principal place for adjusting the emsandbox tarball to suit the needs of specific machine variants. setup.sh and config.sh can fine tune the results but in order to avoid reinventing the wheel, if more than a few machines need similar adjustments to the same files, future versions of packages.conf will collate those into a single configuration parameter available to all.

packages.conf supports:

INCLUDE

Add a comma separated list of package names to the list of packages added to the tarball and installed in the second stage. Currently, debootstrap has problems with multiple repositories so either upload this package to the same repository as your other packages or create an apt-proxy that can serve as a local repository, set it in PROXY and specify a usable mirror for the device in MIRROR.
DEFAULT: empty

SCRIPT

Overrides the default emsandbox suite-script that debootstrap uses to determine the base and required packages and the all important sequence in which the packages can be installed. SCRIPT can be overridden on the emsandbox command line.
DEFAULT: /usr/share/emdebian-tools/emdebian.crossd

MIRROR

Overrides the default emsandbox mirror. This repository will be set in /etc/apt/sources.list and will also be used by debootstrap to obtain all packages for the tarball unless PROXY is also set.
DEFAULT: http://www.emdebian.org/emdebian/

PROXY

Specifies a separate repository to pass to debootstrap that may be local or otherwise not intended for use once the tarball is installed. Use MIRROR to set the same value in debootstrap and /etc/apt/sources.list. If PROXY is specified without MIRROR, the default emsandbox MIRROR (http://www.emdebian.org/emdebian/) will be written into /etc/apt/sources.list.
DEFAULT: empty

TARBALL_NAME

Overrides the default name (emdebian-$ARCH) of the tarball. Do not specify a path here, just a filename with the .tgz suffix.
DEFAULT: emdebian-$ARCH.tgz where $ARCH is specified to emsandbox or as the dpkg-cross default architecture.

SUITE

Not recommended to be changed.
DEFAULT: unstable

Due to limitations in the current debootstrap support, the only way of adding packages to the first stage is by providing a customised suite script. Even if emsandbox migrates to using a tool from Stag to overcome shortcomings in debootstrap, support for packages.conf, setup.sh and config.sh will remain.

TIPS AND HINTS

• Autobuilds can take a long time and downloading the native and cross dependencies can hog your network bandwidth, making it hard to do other work on the packages.
Consider using trickle. From trickle (1):
 $ trickle -u 10 -d 20 em_autobuild -a arm -p -l ...
         
 

Launch an autobuilder, limiting the upload capacity to 10 KB/s, and download capacity at 20 KB/s.

Manual uploads will show up on the autobuilder report page until the .upload file is appended. Use emsource --status package to locate the relevant .upload file and append it to the latest build log, e.g. :
 $ emsource --status gpe-mixer
 ...
 .upload:
 /opt/emdebian/pbuilder/result/g/gpe-mixer/trunk/gpe-mixer_0.42-2em1_arm.upload
 5 files uploaded to buildd.emdebian.org on Sun Sep 21 17:31:19 2008 for 0.42-2em1
 ...
 Checking autobuilder logs in /opt/emdebian/buildd/g/gpe-mixer/trunk/
 /opt/emdebian/buildd/g/gpe-mixer/trunk/gpe-mixer-arm-1221987469.log
 $ cat /opt/emdebian/pbuilder/result/g/gpe-mixer/trunk/gpe-mixer_0.42-2em1_arm.upload >> \
  /opt/emdebian/buildd/g/gpe-mixer/trunk/gpe-mixer-arm-1221987469.log
       
 

The modified build log can be synchronised with the webserver when the next autobuild is run or manually once other manual uploads have been completed.

Scripts to assist in managing the autobuilder are available in the emdebian-qa package and can be found in /usr/share/doc/emdebian-qa/buildd/.

AUTHOR

em_autobuild was written by Neil Williams codehelp@debian.org.

This manual page was written by Neil Williams codehelp@debian.org

SEE ALSO

See also apt-cross (1), em_make (1), emrecent (1), emsource (1), dpkg-cross (1), emdebian-tools (1).

AUTHOR

Emdebian

Neil Williams

Debian and Emdebian developer.