Difference between revisions of "Distcc"
(→code 110: should not experience this at all using the AUR package)
(→Troubleshooting: Add some tooltips about error when using distccd-alarm-arm*)
|Line 186:||Line 186:|
$ export DISTCC_DIR=/tmp/distcc
$ export DISTCC_DIR=/tmp/distcc
== See also ==
== See also ==
Revision as of 08:23, 14 February 2019
distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network to speed up building. It should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile. Further, one can use it together with native Arch build tools such as makepkg.
- 1 Terms
- 2 Getting started
- 3 Configuration
- 4 Compile
- 5 Monitoring progress
- 6 Cross Compiling with distcc
- 7 Troubleshooting
- 8 See also
- The master is the computer which initiates the compilation.
- The slave accepts compilation requests send by the master. One can setup multiple slave systems or just a single one.
Install the package on all participating PCs in the distcc cluster. For other distros, or even operating systems including Windows through using Cygwin, refer to the distcc docs. Be sure to allow traffic on the port on which distcc runs (the default is 3632/tcp), see Category:Firewalls and .
The configuration for the slave machine is stored in
/etc/conf.d/distccd. At a minimum, configure the allowed address ranges in CIDR format:
A nice tool for converting address ranges to CIDR format can be found here: CIDR Utility Tool. Other commandline options can be defined as well. Refer to .
For use with makepkg
/etc/makepkg.conf in the following sections:
- The BUILDENV array will need to have distcc unbanged i.e. list it without exclamation point.
- Uncomment the DISTCC_HOSTS line and add the host name or IP addresses of the slaves. Optionally, follow this with a forward slash and the max number of threads they are to use. The subsequent nodes should be separated by a white space. This list should be ordered from most powerful to least powerful (processing power).
- Adjust the MAKEFLAGS variable to correspond to the number of sum of the number of individual values specified for the max threads per server. In the example below, this is 5+3+3=11.
Example using relevant lines:
BUILDENV=(distcc fakeroot color !ccache check !sign) MAKEFLAGS="-j11" DISTCC_HOSTS="192.168.10.2/5 192.168.10.3/3 192.168.10.4/3"
-march=nativeflag cannot be used in the
CXXFLAGSvariables, otherwise distccd will not distribute work to other machines.
For use without makepkg
The minimal configuration for distcc on the master includes the setting of the available slaves. This can either be done by setting the addresses in the environment variable
DISTCC_HOSTS or in either of the configuration files
Example for setting the slave address using
$ export DISTCC_HOSTS="192.168.10.3,lzo,cpp 192.168.10.4,lzo,cpp"
Example for setting the slave addresses in the hosts configuration file:
Instead of explicitly listing the server addresses one can also use the avahi zeroconf mode. To use this mode
+zeroconf must be in place instead of the server addresses and the distcc daemons on the slaves have to be started using the
--zeroconf option. Note that this option does not support the pump mode!
The examples add the following options to the address:
lzo: Enables LZO compression for this TCP or SSH host (slave).
cpp: Enables distcc-pump mode for this host (slave). Note: the build command must be wrapped in the pump script in order to start the include server.
A description for the pump mode can be found here: distcc's pump mode: A New Design for Distributed C/C++ Compilation
To use distcc-pump mode for a slave, users must start the compilation using the pump script otherwise the compilation will fail.
Compile via makepkg as normal.
To compile a source file using the distcc pump mode, use the following command:
$ pump distcc g++ -c hello_world.cpp
In this case the pump script will execute distcc which in turn calls g++ with "-c hello_world.cpp" as parameter.
To compile a Makefile project, first find out which variables are set by the compiler. For example in gzip-1.6, one can find the following line in the Makefile:
CC = gcc -std=gnu99. Normally the variables are called
CC for C projects and
CXX for C++ projects. To compile the project using distcc it would look like this:
$ wget ftp://ftp.gnu.org/pub/gnu/gzip/gzip-1.6.tar.xz $ tar xf gzip-1.6.tar.xz $ cd gzip-1.6 $ ./configure $ pump make -j2 CC="distcc gcc -std=gnu99"
This example would compile gzip using distcc's pump mode with two compile threads. For the correct
-j setting have a look at What -j level to use?
ships with a cli monitor
distccmon-text and a gtk monitor
distccmon-gnome one can use to check on compilation status.
The cli monitor can run continuously by appending a space followed by integer to the command which corresponds to the number of sec to wait for a repeat query:
$ distccmon-text 3 29291 Preprocess probe_64.c 192.168.10.2 30954 Compile apic_noop.c 192.168.10.2 30932 Preprocess kfifo.c 192.168.10.2 30919 Compile blk-core.c 192.168.10.2 30969 Compile i915_gem_debug.c 192.168.10.2 30444 Compile block_dev.c 192.168.10.3 30904 Compile compat.c 192.168.10.3 30891 Compile hugetlb.c 192.168.10.3 30458 Compile catalog.c 192.168.10.4 30496 Compile ulpqueue.c 192.168.10.4 30506 Compile alloc.c 192.168.10.4
Cross Compiling with distcc
One can use distcc to help cross compile.
- A machine running the target architecture should be used as the master.
- Non-native architecture slaves will help compile but they require the corresponding toolchain to be installed and their distcc to point to it.
Arch Linux ARM
The developers highly recommend using the official project toolchains which should be installed on the x86_64 slave machine(s). Rather than manually managing these, the AUR provides all four toolchains as well as simple systemd service units:
Setup on the slave machine containing the toolchain is identical to #Slaves except that the name of the configuration and systemd service file matches that of the respective package. For example, for armv7h the config file is
/etc/conf.d/distccd-armv7h and the systemd service unit is
Note that each of the toolchains runs on a unique port thus allowing them to co-exist on the slave machine if needed. Be sure to allow traffic to the port on which distcc runs see Category:Firewalls and .
|Target architecture||Distcc Port|
Setup of the master is identical to #Master except, one needs to modify the following two files to define the now non-standard port the slaves are expected to use. Refer to the table above if using the AUR package.
/etc/conf.d/distcc: example on an armv7h machine:
DISTCC_ARGS="--allow 127.0.0.1 --allow 192.168.10.0/24 --port 3635
/etc/makepkg.conf: example on an armv7h machine:
- EmbToolkit: Tool for creating cross compilation tool chain; supports ARM and MIPS architectures; supports building of an LLVM based tool chain
- crosstool-ng: Similar to EmbToolkit; supports more architectures (see website for more information)
- Linaro: Provides tool chains for ARM development
EmbToolkit provides a nice graphical configuration menu (
make xconfig) for configuring the tool chain.
journalctl to find out what was going wrong:
$ journalctl $(which distccd) -e --since "5 min ago"
Adjust log level
By default, distcc will log to
/var/log/messages.log as it goes along. One trick (actually recommended in the distccd manpage) is to log to an alternative file directly. Again, one can locate this in RAM via /tmp. Another trick is to lower to log level of minimum severity of error that will be included in the log file. Useful if only wanting to see error messages rather than an entry for each connection. LEVEL can be any of the standard syslog levels, and in particular critical, error, warning, notice, info, or debug.
Either call distcc with the arguments mentioned here on the master or appended it to DISTCC_ARGS in
/etc/conf.d/distccd on the slaves:
DISTCC_ARGS="--allow 192.168.10.0/24 --log-level error --log-file /tmp/distccd.log"
Limit HDD/SSD usage by relocating $HOME/.distcc
By default, distcc creates
$HOME/.distcc which stores transient relevant info as it serves up work for nodes to compile. This will avoid needless HDD read/writes and is particularly important for SSDs.
$ export DISTCC_DIR=/tmp/distcc
No such file or directory
If you got an error like:
distcc (dcc_execvp) ERROR: failed to exec armv7l-unknown-linux-gnueabihf-g++: No such file or directory
then you are running distccd and not the distccd-alarm-arm-* service. Just stop distccd and start distccd-alarm-arm*.
Monitoring does not work
As distccd-alarm-arm* services use the
nobody user, you can't see what's going on. You are only left with the option to tail the log in /tmp. For example,
tail -f /tmp/distccd-armv7h.log