Difference between revisions of "Distcc"

From ArchWiki
Jump to navigation Jump to search
m (rm from Category:Networking, doesn't fit there)
(Add link to icecream)
 
(91 intermediate revisions by 27 users not shown)
Line 1: Line 1:
 +
{{Lowercase title}}
 +
[[Category:Distributed computing]]
 +
[[Category:Development]]
 
[[Category:Package development]]
 
[[Category:Package development]]
 
[[it:Distcc]]
 
[[it:Distcc]]
[[zh-CN:Distcc]]
+
[[ja:Distcc]]
{{Article summary start}}
+
[[zh-hans:Distcc]]
{{Article summary text|
+
[[ru:Distcc]]
 +
{{Related articles start}}
 +
{{Related|TORQUE}}
 +
{{Related|Slurm}}
 +
{{Related articles end}}
  
'''distcc''' is a program that distributes source code among a number of distcc-servers allowing many machines to compile one program and thus speed up the compilation process. The cool part is you can use it together with pacman/srcpac.}}
+
[[Wikipedia:distcc|distcc]] is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network to speed up building. It should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile. Further, one can use it together with native Arch build tools such as makepkg.
{{Article summary heading|Related}}
 
{{Article summary wiki|TORQUE}}
 
{{Article summary end}}
 
  
Distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network. distcc should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile.
+
== Terms ==
  
==Terms==
+
; master: The master is the computer which initiates the compilation.
{{Note| The terminology used by the software can be a bit counterintuitive in that "the daemon" is the master and "the server(s)" are the slave PC(s) in a distcc cluster.}}
+
; slave: The slave accepts compilation requests send by the master.  One can setup multiple slave systems or just a single one.
  
; distcc daemon: The PC or server that's running distcc to distribute the source code. The daemon itself will compile parts of the source code but will also send other parts to the hosts defined in ''DISTCC_HOSTS''. 
 
  
; distcc server: The PC or server compiling the source code it gets from the daemon. When it's done compiling, it sends back the object code (i.e. compiled source code) to the daemon, which in turn sends back some more source code (if there's any left to compile).
+
== Getting started ==
  
==Getting started==
+
[[Install]] the {{pkg|distcc}} package on all participating  PCs in the distcc cluster.  For other distros, or even operating systems including Windows through using Cygwin, refer to the [http://distcc.samba.org/doc.html distcc docs].  Be sure to allow traffic on the port on which distcc runs (the default is 3632/tcp), see [[:Category:Firewalls]] and {{man|1|distcc}}.
  
[[pacman|Install]] the {{pkg|distcc}} package from [[Official Repositories|official repositories]] on all PCs in the cluster:
+
== Configuration ==
  
For other distros, or even OSes including Windows through using Cygwin, refer to the [http://distcc.samba.org/doc.html distcc docs].
+
=== Slaves ===
  
==Configuration==
+
The configuration for the slave machine is stored in {{ic|/etc/conf.d/distccd}}. At a minimum, configure the allowed address ranges in [[wikipedia:Classless_Inter-Domain_Routing|CIDR]] format:
===Both Daemon and Server(s)===
 
Edit {{ic|/etc/conf.d/distccd}} and modify the only uncommented line with the correct IP address or range of your daemon or of your entire subnet:
 
DISTCC_ARGS="--user nobody --allow 192.168.0.0/24"
 
  
===Daemon Only===
+
DISTCC_ARGS="--allow 192.168.10.0/24"
  
Edit {{ic|/etc/makepkg.conf}} in the following three sections:
+
A nice tool for converting address ranges to CIDR format can be found here: [http://www.ipaddressguide.com/cidr CIDR Utility Tool]Other commandline options can be defined as wellRefer to {{man|1|distcc}}.  
#BUILDENV has distcc unbanged i.e. without exclamation point.
 
#Uncomment the ''DISTCC_HOSTS'' line and add the IP addresses of the servers (slaves) then a slash and the number of threads they are to useThe subsequent IP address/threads should be separated by a white spaceThis list is ordered from most powerful to least powerful (processing power).
 
#Adjust the MAKEFLAGS variable to correspond to the number of sum of the number of individual values specified for the max threads per server.  In the example below, this is 5+3+3=11.  If users specify more than this sum, the extra theoretical thread(s) will be blocked by distcc and appear as such in monitoring utils such as ''distccmon-text'' described below.
 
  
{{Note|It is common practice to define the number of threads as the number of physical core+hyperhtreaded cores (if they exist) plus 1Do this on a per-server basis, NOT in the MAKEFLAGS!}}
+
[[Start]] {{ic|distccd.service}} on every participating slave. To have {{ic|distccd.service}} start at boot-up, [[enable]] it.
 +
 
 +
=== Master ===
 +
==== For use with makepkg ====
 +
 
 +
Edit {{ic|/etc/makepkg.conf}} in the following sections:
 +
 
 +
# The BUILDENV array will need to have ''distcc'' unbanged i.e. list it without exclamation point.
 +
# Uncomment the ''DISTCC_HOSTS'' line and add the host name or IP addresses of the slaves. Optionally, follow this with a forward slash and the max number of threads they are to useThe subsequent nodes should be separated by a white space.  This list should be ordered from most powerful to least powerful (processing power).
 +
# Adjust the MAKEFLAGS variable to correspond to the number of sum of the number of individual values specified for the max threads per server.  In the example below, this is 5+3+3=11.
  
 
Example using relevant lines:
 
Example using relevant lines:
  BUILDENV=(distcc fakeroot color !ccache !check)
+
 
 +
  BUILDENV=(distcc fakeroot color !ccache check !sign)
 
  MAKEFLAGS="-j11"
 
  MAKEFLAGS="-j11"
  DISTCC_HOSTS="192.168.0.2/5 192.168.0.3/3 192.168.0.4/3"
+
  DISTCC_HOSTS="192.168.10.2/5 192.168.10.3/3 192.168.10.4/3"
 +
 
 +
{{Warning|The {{ic|1=-march=native}} flag cannot be used in the {{ic|CFLAGS}} and {{ic|CXXFLAGS}} variables, otherwise distccd will not distribute work to other machines.}}
 +
 
 +
==== For use without makepkg ====
  
If users wish to use distcc through SSH, add an "@" symbol in front of the IP address in this section. If key-based auth is not setup on the systems, set the DISTCC_SSH variable to ignore checking for authenticated hosts, i.e. DISTCC_SSH="ssh -i"
+
The minimal configuration for distcc on the master includes the setting of the available slaves. This can either be done by setting the addresses in the environment variable {{ic|DISTCC_HOSTS}} or in either of the configuration files {{ic|$DISTCC_HOSTS}}, {{ic|$DISTCC_DIR/hosts}}, {{ic|~/.distcc/hosts}} or {{ic|/etc/distcc/hosts}}.
  
==Compile==
+
Example for setting the slave address using {{ic|DISTCC_HOSTS}}:
Start the distcc daemon on every participating machine:
 
# systemctl start distccd
 
  
To have {{ic|distccd}} start at boot-up, run this on every participating machine:
+
$ export DISTCC_HOSTS="192.168.10.3,lzo,cpp 192.168.10.4,lzo,cpp"
# systemctl enable distccd
 
  
Compile via makepkg as normal.
+
{{Note|This is a white space separated list.}}
  
==Monitoring Progress==
+
Example for setting the slave addresses in the hosts configuration file:
Progress can be monitored via several methods.
 
#distccmon-text
 
#tailing log file
 
  
Invoke distccmon-text to check on compilation status:
+
{{hc|~/.distcc/hosts|
{{bc|$ distccmon-text
+
192.168.10.3,lzo,cpp 192.168.10.4,lzo,cpp
29291 Preprocess  probe_64.c                                192.168.0.2[0]
 
30954 Compile    apic_noop.c                                192.168.0.2[0]
 
30932 Preprocess  kfifo.c                                    192.168.0.2[0]
 
30919 Compile    blk-core.c                                192.168.0.2[1]
 
30969 Compile    i915_gem_debug.c                          192.168.0.2[3]
 
30444 Compile    block_dev.c                                192.168.0.3[1]
 
30904 Compile    compat.c                                  192.168.0.3[2]
 
30891 Compile    hugetlb.c                                  192.168.0.3[3]
 
30458 Compile    catalog.c                                  192.168.0.4[0]
 
30496 Compile    ulpqueue.c                                192.168.0.4[2]
 
30506 Compile    alloc.c                                    192.168.0.4[0]
 
 
}}
 
}}
  
One can have this program run continuously by using watch or by appending a space followed by integer to the command which corresponds to the number of sec to wait for a repeat query:
+
Instead of explicitly listing the server addresses one can also use the avahi zeroconf mode. To use this mode {{ic|+zeroconf}} must be in place instead of the server addresses and the distcc daemons on the slaves have to be started using the {{ic|--zeroconf}} option. Note that this option does not support the pump mode!
$ watch distccmon-text
+
 
or
+
The examples add the following options to the address:
$ distccmon-text 2
+
 
 +
* {{ic|lzo}}: Enables LZO compression for this TCP or SSH host (slave).
 +
* {{ic|cpp}}: Enables distcc-pump mode for this host (slave). Note: the build command must be wrapped in the pump script in order to start the include server.
  
One can also simply tail {{ic|/var/log/messages.log}} on daemon:
+
A description for the pump mode can be found here: [http://google-opensource.blogspot.de/2008/08/distccs-pump-mode-new-design-for.html distcc's pump mode: A New Design for Distributed C/C++ Compilation ]
# tail -f /var/log/messages.log
 
  
=="Cross Compiling" with Distcc==
+
To use distcc-pump mode for a slave, users must start the compilation using the pump script otherwise the compilation will fail.
There are currently two method from which to select to have the ability of distcc distribution of tasks over a cluster building i686 packages from a native x86_64 environment.  Neither is ideal, but to date, there are the only two methods documented on the wiki.
 
  
An ideal setup is one that uses the unmodified ARCH packages for distccd running only once one each node regardless of building from the native environment or from within a chroot AND one that works with makepkg.  Again, this Utopian setup is not currently known.
+
== Compile ==
 +
=== With makepkg ===
  
A [https://bbs.archlinux.org/viewtopic.php?id=129762 discussion thread] has been started on the topic; feel free to contribute.
+
Compile via makepkg as normal.
  
=== Chroot Method (Preferred) ===
+
=== Without makepkg ===
{{Note|This method works, but is not very elegant requiring duplication of distccd on all nodes AND need to have a 32-bit chroots on all nodes.}}
 
  
Assuming the user has a [[Install_bundled_32-bit_system_in_Arch64|32-bit chroot]] setup and configured on '''each node''' of the distcc cluster, the strategy is to have two separate instances of distccd running on different ports on each node -- one runs in the native x86_64 environment and the other in the x86 chroot on a modified port.  Start makepkg via a [[Install_bundled_32-bit_system_in_Arch64#Executing_32-bit_applications_from_a_64-bit_environment|schroot command]] invoking makepkg.
+
To compile a source file using the distcc pump mode, use the following command:
  
==== Add port numbers to DISTCC_HOSTS on the i686 chroot ====
+
$ pump distcc g++ -c hello_world.cpp
Append the port number defined eariler (3692) to each of the hosts in {{ic|/opt/arch32/etc/makepkg.conf}} as follows:
 
  
DISTCC_HOSTS="192.168.1.101/5:3692 192.168.1.102/5:3692 192.168.1.103/3:3692"
+
In this case the pump script will execute distcc which in turn calls g++ with "-c hello_world.cpp" as parameter.
  
{{Note|This only needs to be setup on the "master" i686 chrootWhere "master" is defined as the one from which the compilation will take place.}}
+
To compile a Makefile project, first find out which variables are set by the compiler. For example in gzip-1.6, one can find the following line in the Makefile: {{ic|1=CC = gcc -std=gnu99}}. Normally the variables are called {{ic|CC}} for C projects and {{ic|CXX}} for C++ projects. To compile the project using distcc it would look like this:
  
==== Invoke makepkg from the Native Environment ====
+
$ wget ftp://ftp.gnu.org/pub/gnu/gzip/gzip-1.6.tar.xz
 +
$ tar xf gzip-1.6.tar.xz
 +
$ cd gzip-1.6
 +
$ ./configure
 +
$ pump make -j2 CC="distcc gcc -std=gnu99"
  
Setup [[Install_bundled_32-bit_system_in_Arch64#Executing_32-bit_applications_from_a_64-bit_environment|schroot]] on the native x86_64 environment.  Invoke makepkg to build an i686 package from the native x86_64 environment, simply by:
+
This example would compile gzip using distcc's pump mode with two compile threads. For the correct {{ic|-j}} setting have a look at [https://cdn.rawgit.com/distcc/distcc/master/doc/web/faq.html What -j level to use?]
  
$ schroot -p -- makepkg -src
+
== Monitoring progress ==
 +
{{Pkg|distcc}} ships with a cli monitor {{ic|distccmon-text}} and a gtk monitor {{ic|distccmon-gnome}} one can use to check on compilation status.
  
=== Multilib GCC Method (Not Recommended) ===
+
The cli monitor can run continuously by appending a space followed by integer to the command which corresponds to the number of sec to wait for a repeat query:
{{Warning|Errors have been reported when using this method to build the i686 linux package from a native x86_64 system!  The chroot method is preferred and has been verified to work building the kernel packages.}}
 
  
Edit {{ic|/etc/pacman.conf}} and uncomment the [multilib] repository:
+
{{bc|$ distccmon-text 3
 +
29291 Preprocess  probe_64.c                                192.168.10.2[0]
 +
30954 Compile    apic_noop.c                                192.168.10.2[0]
 +
30932 Preprocess  kfifo.c                                    192.168.10.2[0]
 +
30919 Compile    blk-core.c                                192.168.10.2[1]
 +
30969 Compile    i915_gem_debug.c                          192.168.10.2[3]
 +
30444 Compile    block_dev.c                                192.168.10.3[1]
 +
30904 Compile    compat.c                                  192.168.10.3[2]
 +
30891 Compile    hugetlb.c                                  192.168.10.3[3]
 +
30458 Compile    catalog.c                                  192.168.10.4[0]
 +
30496 Compile    ulpqueue.c                                192.168.10.4[2]
 +
30506 Compile    alloc.c                                    192.168.10.4[0]
 +
}}
  
  [multilib]
+
== Cross Compiling with distcc ==
Include = /etc/pacman.d/mirrorlist
+
One can use distcc to help cross compile.  
 +
* A machine running the target architecture should be used as the master.
 +
* Non-native architecture slaves will help compile but they require the corresponding toolchain to be installed and their distcc to point to it.  
  
[[pacman|Install]] {{pkg|gcc-multilib}} from the [[Official Repositories|official repositories]].
+
=== Arch Linux ARM ===
 +
==== Slaves ====
 +
The developers ''highly'' recommend using the official project [https://archlinuxarm.org/wiki/Distcc_Cross-Compiling toolchains] which should be installed on the x86_64 slave machine(s).  Rather than manually managing these, the [[AUR]] provides all four toolchains as well as simple systemd service units:
 +
* {{AUR|distccd-alarm-armv5}}
 +
* {{AUR|distccd-alarm-armv6h}}
 +
* {{AUR|distccd-alarm-armv7h}}
 +
* {{AUR|distccd-alarm-armv8}}
  
Compile packages on x86_64 for i686 is as easy as adding the following lines to {{ic|$HOME/.makepkg.conf}}
+
Setup on the slave machine containing the toolchain is identical to [[#Slaves]] except that the name of the configuration and systemd service file matches that of the respective package.  For example, for armv7h the config file is {{ic|/etc/conf.d/distccd-armv7h}} and the systemd service unit is {{ic|distccd-armv7h.service}}.
CARCH="i686"
 
CHOST="i686-pc-linux-gnu"
 
CFLAGS="-march=i686 -O2 -pipe -m32"
 
CXXFLAGS="${CFLAGS}"
 
  
and invoking makepkg via the following
+
Note that each of the toolchains runs on a unique port thus allowing them to co-exist on the slave machine if needed. Be sure to allow traffic to the port on which distcc runs see [[:Category:Firewalls]] and {{man|1|distcc}}.
  $ linux32 makepkg -src
 
  
Remember to remove or modify {{ic|$HOME/.makepkg.conf}} when finished compiling i686 packages!
+
{| class="wikitable" align="center"
 +
|-
 +
! Target architecture !! Distcc Port
 +
|-
 +
| ''armv5'' || 3633
 +
|-
 +
| ''armv6h'' || 3634
 +
|-
 +
| ''armv7h'' || 3635
 +
|-
 +
| ''armv8h/aarch64'' || 3636
 +
|-
 +
|}
  
==Tips/Tricks==
+
==== Master ====
===Limit HDD/SSD usage===
+
Setup of the master is identical to [[#Master]] except, one needs to modify the following two files to define the now non-standard port the slaves are expected to use.  Refer to the table above if using the AUR package.
====Relocate $HOME/.distcc====
+
# {{ic|/etc/conf.d/distcc}}: example on an armv7h machine: {{ic|1=DISTCC_ARGS="--allow 127.0.0.1 --allow 192.168.10.0/24 --port 3635}}
By default, distcc creates {{ic|$HOME/.distcc}} which stores transient relevant info as it serves up work for nodes to compile. Create a directory named ''.distcc'' in RAM such as /tmp and soft link to it in $HOME. This will avoid needless HDD read/writes and is particularly important for SSDs.
+
# {{ic|/etc/makepkg.conf}}: example on an armv7h machine: {{ic|1=DISTCC_HOSTS="192.168.10.2/5:3635 192.168.10.3/5:3635"}}
  
$ mv $HOME/.distcc /tmp
+
=== Additional toolchains ===
$ ln -s /tmp/.distcc $HOME/.distcc
+
* [https://embtoolkit.org/ EmbToolkit]: Tool for creating cross compilation tool chain; supports ARM and MIPS architectures; supports building of an LLVM based tool chain
 +
* [http://crosstool-ng.org/ crosstool-ng]: Similar to EmbToolkit; supports more architectures (see website for more information)
 +
* [https://www.linaro.org/downloads/ Linaro]: Provides tool chains for ARM development
  
Use systemd to re-create this directory on a reboot (the soft link will remain until it is manually removed like any other file):
+
The {{ic|EmbToolkit}} provides a nice graphical configuration menu ({{ic|make xconfig}}) for configuring the tool chain.
  
Create the following tmpfile where "X" is the letter for the SSD device.
+
== Troubleshooting ==
  
{{hc| /etc/tmpfiles.d/tmpfs-create.conf |<nowiki>
+
=== Journalctl ===
d /tmp/.distcc 0755 <username> users -
 
</nowiki>}}
 
  
====Adjust log level====
+
Use {{ic|journalctl}} to find out what was going wrong:
 +
 
 +
$ journalctl $(which distccd) -e --since "5 min ago"
 +
 
 +
=== Adjust log level ===
  
 
By default, distcc will log to {{ic|/var/log/messages.log}} as it goes along.  One trick (actually recommended in the distccd manpage) is to log to an alternative file directly.  Again, one can locate this in RAM via /tmp.  Another trick is to lower to log level of minimum severity of error that will be included in the log file.  Useful if only wanting to see error messages rather than an entry for each connection.  LEVEL can be any of the  standard syslog levels, and in particular critical, error, warning, notice, info, or debug.
 
By default, distcc will log to {{ic|/var/log/messages.log}} as it goes along.  One trick (actually recommended in the distccd manpage) is to log to an alternative file directly.  Again, one can locate this in RAM via /tmp.  Another trick is to lower to log level of minimum severity of error that will be included in the log file.  Useful if only wanting to see error messages rather than an entry for each connection.  LEVEL can be any of the  standard syslog levels, and in particular critical, error, warning, notice, info, or debug.
  
Both of these lines are to be '''appended''' to DISTCC_ARGS in {{ic|/etc/conf.d/distccd}}
+
Either call distcc with the arguments mentioned here on the master or appended it to DISTCC_ARGS in {{ic|/etc/conf.d/distccd}} on the slaves:
 +
 
 +
DISTCC_ARGS="--allow 192.168.10.0/24 --log-level error --log-file /tmp/distccd.log"
 +
 
 +
=== Limit HDD/SSD usage by relocating $HOME/.distcc ===
 +
 
 +
By default, distcc creates {{ic|$HOME/.distcc}} which stores transient relevant info as it serves up work for nodes to compile. This will avoid needless HDD read/writes and is particularly important for SSDs.
 +
 
 +
$ export DISTCC_DIR=/tmp/distcc
 +
 
 +
=== For distccd-alarm ===
 +
==== No such file or directory ====
 +
Errors similar to the following indicate that the user is mistakenly running the distccd service provided by {{pkg|distcc}} and NOT provided by the distccd-alarm packages (ie {{AUR|distccd-alarm-armv5}}, {{AUR|distccd-alarm-armv6h}}, {{AUR|distccd-alarm-armv7h}}, or {{AUR|distccd-alarm-armv8}}.)
 +
 
 +
Be sure to start the correct service for the target architecture.
 +
 
 +
{{bc|distcc[25479] (dcc_execvp) ERROR: failed to exec armv7l-unknown-linux-gnueabihf-g++: No such file or directory
 +
}}
 +
 
 +
==== Monitoring does not work ====
 +
Since the distccd-alarm-arm* services use the {{ic|nobody}} user, {{ic|/usr/bin/distccmon-text}} does not work to monitor compilation.  One can inspect the output of the actually distcc log to troubleshoot if needed, {{ic|tail -f /tmp/distccd-armv7h.log}} for example.
 +
 
 +
== See also ==
  
DISTCC_ARGS="--user nobody --allow 192.168.0.0/24 --log-level error --log-file /tmp/distccd.log"
+
* https://github.com/distcc/distcc
  
==Failure work with CMake or other tools==
+
* {{AUR|icecream}} - An easier to configure fork of distcc that some find brings notably better results and improved utilisation of available cores.
CMake sometimes pass [http://gcc.gnu.org/wiki/Response_Files "response file"] to gcc, but the distcc cannot deal with it.
 
There is a [http://code.google.com/p/distcc/issues/detail?id=85&q=response patch file], but it has not been applied to upstream code.
 
If you encounter this problem, you can path this file or use the {{AUR|distcc-rsp}} package in the [[Arch User Repository|AUR]].
 

Latest revision as of 20:24, 21 February 2019

distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network to speed up building. It should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile. Further, one can use it together with native Arch build tools such as makepkg.

Terms

master
The master is the computer which initiates the compilation.
slave
The slave accepts compilation requests send by the master. One can setup multiple slave systems or just a single one.


Getting started

Install the distcc package on all participating PCs in the distcc cluster. For other distros, or even operating systems including Windows through using Cygwin, refer to the distcc docs. Be sure to allow traffic on the port on which distcc runs (the default is 3632/tcp), see Category:Firewalls and distcc(1).

Configuration

Slaves

The configuration for the slave machine is stored in /etc/conf.d/distccd. At a minimum, configure the allowed address ranges in CIDR format:

DISTCC_ARGS="--allow 192.168.10.0/24"

A nice tool for converting address ranges to CIDR format can be found here: CIDR Utility Tool. Other commandline options can be defined as well. Refer to distcc(1).

Start distccd.service on every participating slave. To have distccd.service start at boot-up, enable it.

Master

For use with makepkg

Edit /etc/makepkg.conf in the following sections:

  1. The BUILDENV array will need to have distcc unbanged i.e. list it without exclamation point.
  2. Uncomment the DISTCC_HOSTS line and add the host name or IP addresses of the slaves. Optionally, follow this with a forward slash and the max number of threads they are to use. The subsequent nodes should be separated by a white space. This list should be ordered from most powerful to least powerful (processing power).
  3. Adjust the MAKEFLAGS variable to correspond to the number of sum of the number of individual values specified for the max threads per server. In the example below, this is 5+3+3=11.

Example using relevant lines:

BUILDENV=(distcc fakeroot color !ccache check !sign)
MAKEFLAGS="-j11"
DISTCC_HOSTS="192.168.10.2/5 192.168.10.3/3 192.168.10.4/3"
Warning: The -march=native flag cannot be used in the CFLAGS and CXXFLAGS variables, otherwise distccd will not distribute work to other machines.

For use without makepkg

The minimal configuration for distcc on the master includes the setting of the available slaves. This can either be done by setting the addresses in the environment variable DISTCC_HOSTS or in either of the configuration files $DISTCC_HOSTS, $DISTCC_DIR/hosts, ~/.distcc/hosts or /etc/distcc/hosts.

Example for setting the slave address using DISTCC_HOSTS:

$ export DISTCC_HOSTS="192.168.10.3,lzo,cpp 192.168.10.4,lzo,cpp"
Note: This is a white space separated list.

Example for setting the slave addresses in the hosts configuration file:

~/.distcc/hosts
192.168.10.3,lzo,cpp 192.168.10.4,lzo,cpp

Instead of explicitly listing the server addresses one can also use the avahi zeroconf mode. To use this mode +zeroconf must be in place instead of the server addresses and the distcc daemons on the slaves have to be started using the --zeroconf option. Note that this option does not support the pump mode!

The examples add the following options to the address:

  • lzo: Enables LZO compression for this TCP or SSH host (slave).
  • cpp: Enables distcc-pump mode for this host (slave). Note: the build command must be wrapped in the pump script in order to start the include server.

A description for the pump mode can be found here: distcc's pump mode: A New Design for Distributed C/C++ Compilation

To use distcc-pump mode for a slave, users must start the compilation using the pump script otherwise the compilation will fail.

Compile

With makepkg

Compile via makepkg as normal.

Without makepkg

To compile a source file using the distcc pump mode, use the following command:

$ pump distcc g++ -c hello_world.cpp

In this case the pump script will execute distcc which in turn calls g++ with "-c hello_world.cpp" as parameter.

To compile a Makefile project, first find out which variables are set by the compiler. For example in gzip-1.6, one can find the following line in the Makefile: CC = gcc -std=gnu99. Normally the variables are called CC for C projects and CXX for C++ projects. To compile the project using distcc it would look like this:

$ wget ftp://ftp.gnu.org/pub/gnu/gzip/gzip-1.6.tar.xz
$ tar xf gzip-1.6.tar.xz
$ cd gzip-1.6
$ ./configure
$ pump make -j2 CC="distcc gcc -std=gnu99"

This example would compile gzip using distcc's pump mode with two compile threads. For the correct -j setting have a look at What -j level to use?

Monitoring progress

distcc ships with a cli monitor distccmon-text and a gtk monitor distccmon-gnome one can use to check on compilation status.

The cli monitor can run continuously by appending a space followed by integer to the command which corresponds to the number of sec to wait for a repeat query:

$ distccmon-text 3
29291 Preprocess  probe_64.c                                 192.168.10.2[0]
30954 Compile     apic_noop.c                                192.168.10.2[0]
30932 Preprocess  kfifo.c                                    192.168.10.2[0]
30919 Compile     blk-core.c                                 192.168.10.2[1]
30969 Compile     i915_gem_debug.c                           192.168.10.2[3]
30444 Compile     block_dev.c                                192.168.10.3[1]
30904 Compile     compat.c                                   192.168.10.3[2]
30891 Compile     hugetlb.c                                  192.168.10.3[3]
30458 Compile     catalog.c                                  192.168.10.4[0]
30496 Compile     ulpqueue.c                                 192.168.10.4[2]
30506 Compile     alloc.c                                    192.168.10.4[0]

Cross Compiling with distcc

One can use distcc to help cross compile.

  • A machine running the target architecture should be used as the master.
  • Non-native architecture slaves will help compile but they require the corresponding toolchain to be installed and their distcc to point to it.

Arch Linux ARM

Slaves

The developers highly recommend using the official project toolchains which should be installed on the x86_64 slave machine(s). Rather than manually managing these, the AUR provides all four toolchains as well as simple systemd service units:

Setup on the slave machine containing the toolchain is identical to #Slaves except that the name of the configuration and systemd service file matches that of the respective package. For example, for armv7h the config file is /etc/conf.d/distccd-armv7h and the systemd service unit is distccd-armv7h.service.

Note that each of the toolchains runs on a unique port thus allowing them to co-exist on the slave machine if needed. Be sure to allow traffic to the port on which distcc runs see Category:Firewalls and distcc(1).

Target architecture Distcc Port
armv5 3633
armv6h 3634
armv7h 3635
armv8h/aarch64 3636

Master

Setup of the master is identical to #Master except, one needs to modify the following two files to define the now non-standard port the slaves are expected to use. Refer to the table above if using the AUR package.

  1. /etc/conf.d/distcc: example on an armv7h machine: DISTCC_ARGS="--allow 127.0.0.1 --allow 192.168.10.0/24 --port 3635
  2. /etc/makepkg.conf: example on an armv7h machine: DISTCC_HOSTS="192.168.10.2/5:3635 192.168.10.3/5:3635"

Additional toolchains

  • EmbToolkit: Tool for creating cross compilation tool chain; supports ARM and MIPS architectures; supports building of an LLVM based tool chain
  • crosstool-ng: Similar to EmbToolkit; supports more architectures (see website for more information)
  • Linaro: Provides tool chains for ARM development

The EmbToolkit provides a nice graphical configuration menu (make xconfig) for configuring the tool chain.

Troubleshooting

Journalctl

Use journalctl to find out what was going wrong:

$ journalctl $(which distccd) -e --since "5 min ago"

Adjust log level

By default, distcc will log to /var/log/messages.log as it goes along. One trick (actually recommended in the distccd manpage) is to log to an alternative file directly. Again, one can locate this in RAM via /tmp. Another trick is to lower to log level of minimum severity of error that will be included in the log file. Useful if only wanting to see error messages rather than an entry for each connection. LEVEL can be any of the standard syslog levels, and in particular critical, error, warning, notice, info, or debug.

Either call distcc with the arguments mentioned here on the master or appended it to DISTCC_ARGS in /etc/conf.d/distccd on the slaves:

DISTCC_ARGS="--allow 192.168.10.0/24 --log-level error --log-file /tmp/distccd.log"

Limit HDD/SSD usage by relocating $HOME/.distcc

By default, distcc creates $HOME/.distcc which stores transient relevant info as it serves up work for nodes to compile. This will avoid needless HDD read/writes and is particularly important for SSDs.

$ export DISTCC_DIR=/tmp/distcc

For distccd-alarm

No such file or directory

Errors similar to the following indicate that the user is mistakenly running the distccd service provided by distcc and NOT provided by the distccd-alarm packages (ie distccd-alarm-armv5AUR, distccd-alarm-armv6hAUR, distccd-alarm-armv7hAUR, or distccd-alarm-armv8AUR.)

Be sure to start the correct service for the target architecture.

distcc[25479] (dcc_execvp) ERROR: failed to exec armv7l-unknown-linux-gnueabihf-g++: No such file or directory

Monitoring does not work

Since the distccd-alarm-arm* services use the nobody user, /usr/bin/distccmon-text does not work to monitor compilation. One can inspect the output of the actually distcc log to troubleshoot if needed, tail -f /tmp/distccd-armv7h.log for example.

See also

  • icecreamAUR - An easier to configure fork of distcc that some find brings notably better results and improved utilisation of available cores.