distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network to speed up building. It should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile. Further, one can use it together with native Arch build tools such as makepkg.
- 1 Terms
- 2 Getting started
- 3 Configuration
- 4 Compile
- 5 Monitoring progress
- 6 Cross compiling with distcc
- 7 Troubleshooting
- 8 See also
- The client is the computer initiating the compilation.
- The volunteer is the computer accepting compilation requests sent by the client. One can setup multiple volunteers or just a single one.
Install the package on all participating PCs in the distcc cluster. For other distros, or even operating systems including Windows through using Cygwin, refer to the distcc docs or the included man pages and . Be sure to allow traffic on the port on which distcc runs (the default is 3632/tcp), see Category:Firewalls.
The configuration for the volunteer is stored in
/etc/conf.d/distccd. At a minimum, add the --allow-private switch which covers a number of ipv4 private network ranges. Logging to a file is also nice for troubleshooting if needed.
DISTCC_ARGS="--allow-private --log-file /tmp/distccd.log"
If multiple interfaces are present on the machine, consider passing the --listen ADDRESS option as well. Other options can be defined. Refer to.
For use with makepkg
/etc/makepkg.conf in the following sections:
- The BUILDENV array will need to have distcc unbanged i.e. list it without exclamation point.
- Uncomment the DISTCC_HOSTS line and add the host name or IP addresses of the volunteers. Optionally, follow this with a forward slash and the max number of threads they are to use. The subsequent nodes should be separated by a white space. This list should be ordered from most powerful to least powerful (processing power).
- Adjust the MAKEFLAGS variable to correspond roughly twice the number max threads per server. In the example below, this is 2x(9+5+5+3)=44.
Example using relevant lines:
BUILDENV=(distcc fakeroot color !ccache check !sign) MAKEFLAGS="-j44" DISTCC_HOSTS="localhost/9 192.168.10.2/5 192.168.10.3/5 192.168.10.4/3"
-march=nativeflag cannot be used in the
CXXFLAGSvariables, otherwise distccd will not distribute work to other machines.
For use without makepkg
The minimal configuration for distcc on the client includes the setting of the available volunteers and re-defining the PATH.
$ export DISTCC_HOSTS="localhost/9 192.168.10.2/5 192.168.10.3/5 192.168.10.4/3" $ export PATH="/usr/lib/distcc/bin:$PATH"
Compile via makepkg as normal.
After exporting the two variables describe in #For use without makepkg, compile as usual but append CC= and CXX= lines to your build:
$ make -j44 CC=distcc CXX=distcc
Use the following CMake options to build a CMake-based project with distcc:
$ cmake -DCMAKE_C_COMPILER_LAUNCHER=distcc -DCMAKE_CXX_COMPILER_LAUNCHER=distcc ...
ships with a cli monitor
distccmon-text and a gtk monitor
distccmon-gnome one can use to check on compilation status.
The cli monitor can run continuously by appending a space followed by integer to the command which corresponds to the number of sec to wait for a repeat query:
$ distccmon-text 3 29291 Preprocess probe_64.c 192.168.10.2 30954 Compile apic_noop.c 192.168.10.2 30932 Preprocess kfifo.c 192.168.10.2 30919 Compile blk-core.c 192.168.10.2 30969 Compile i915_gem_debug.c 192.168.10.2 30444 Compile block_dev.c 192.168.10.3 30904 Compile compat.c 192.168.10.3 30891 Compile hugetlb.c 192.168.10.3 30458 Compile catalog.c 192.168.10.4 30496 Compile ulpqueue.c 192.168.10.4 30506 Compile alloc.c 192.168.10.4
Cross compiling with distcc
One can use distcc to help cross compile.
- A machine running the target architecture must be used as the client.
- Non-native architecture volunteers will help compile but they require the corresponding toolchain to be installed and their distcc to point to it.
Arch Linux ARM as clients (x86_64 as volunteers)
This section details how to use Arch Linux (x86_64) volunteers to help an Arch ARM device cross-compile. Significant speed gains can be realized with just a single x86_64 machine helping an ARM device compile.
The Arch ARM developers highly recommend using the official project toolchains which should be installed on the x86_64 volunteer(s). Rather than manually managing these, the AUR provides all four toolchains as well as simple systemd service units:
Setup on the volunteer containing the toolchain is identical to #Volunteers except that the name of the configuration and systemd service file matches that of the respective package. For example, for armv7h the config file is
/etc/conf.d/distccd-armv7h and the systemd service unit is
Note that each of the toolchains runs on a unique port thus allowing all four of them to co-exist on the volunteer if needed. Be sure to allow traffic to the port on which distcc runs see Category:Firewalls and .
|Target architecture||Distcc Port|
The easiest method to setup the Arch ARM client is to use the
/etc/conf.d/distccd-armv7h and change the defaults therein. When ready to build, enable
distccd-armv7h.service and compile.
If one would rather setup the client without using the AUR package mentioned above, manual setup of the client is identical to #Client except, one needs to modify the following two files to define the now non-standard port the volunteers are expected to use. Refer to the table above if using the AUR package.
/etc/conf.d/distcc: example on an armv7h machine:
DISTCC_ARGS="--allow 127.0.0.1 --allow 192.168.10.0/24 --port 3635
/etc/makepkg.conf: example on an armv7h machine:
Arch Linux (x86_64) as clients (Arch ARM as volunteers)
This section details how to use Arch ARM volunteers to help an x86_64 client cross-compile.
Setup of the client is identical to #Client with distcc running on the standard port 3632.
AUR will provide a toolchain to install on the Arch ARM devices to enable cross compilation.
- EmbToolkit[dead link 2020-03-29 ⓘ]: Tool for creating cross compilation tool chain; supports ARM and MIPS architectures; supports building of an LLVM based tool chain
- crosstool-ng: Similar to EmbToolkit; supports more architectures (see website for more information)
- Linaro: Provides tool chains for ARM development
EmbToolkit provides a nice graphical configuration menu (
make xconfig) for configuring the tool chain.
Quirks compiling the Arch Linux kernel package
If building the kernel from the official PKGBUILD (or many from the AUR), one needs to edit the PKGBUILD to pass the CC=distcc and CXX=distcc variables as part of the build command:
make bzImage modules htmldocs CC=distcc CXX=distcc
Failure to do this will result in distcc not working during the build. See FS#64275.
Quirks compiling chromium
journalctl to find out what was going wrong:
$ journalctl $(which distccd) -e --since "5 min ago"
Adjust log level
By default, distcc will log to
/var/log/messages.log as it goes along. One trick (actually recommended in the distccd manpage) is to log to an alternative file directly. Again, one can locate this in RAM via /tmp. Another trick is to lower to log level of minimum severity of error that will be included in the log file. Useful if only wanting to see error messages rather than an entry for each connection. LEVEL can be any of the standard syslog levels, and in particular critical, error, warning, notice, info, or debug.
Either call distcc with the arguments mentioned here on the client or appended it to DISTCC_ARGS in
/etc/conf.d/distccd on the volunteers:
DISTCC_ARGS="--allow 192.168.10.0/24 --log-level error --log-file /tmp/distccd.log"
Limit HDD/SSD usage by relocating $HOME/.distcc
By default, distcc creates
$HOME/.distcc which stores transient relevant info as it serves up work for nodes to compile. This will avoid needless HDD read/writes and is particularly important for SSDs.
$ export DISTCC_DIR=/tmp/distcc
No such file or directory
Errors similar to the following indicate that the user is mistakenly running the distccd service provided byand NOT provided by the distccd-alarm packages (ie AUR, AUR, AUR, or AUR.)
Be sure to start the correct service for the target architecture.
distcc (dcc_execvp) ERROR: failed to exec armv7l-unknown-linux-gnueabihf-g++: No such file or directory
Avahi-daemon stops publishing when distccd.service starts
distccd.service as a service might cause
avahi-daemon to stop working. This can be mitigated by making sure
avahi-daemon.service starts after
distccd.service by editing
avahi-daemon.service's unit file (see Systemd#Editing provided units) and adding
After=distccd.service at the end of the
... [Unit] Description=Avahi mDNS/DNS-SD Stack Requires=avahi-daemon.socket After=distccd.service ...
- AUR - An easier to configure fork of distcc.