Difference between revisions of "Modprobed-db"

From ArchWiki
Jump to: navigation, search
m (grammar typo)
(Installation and Setup: 2.5 years old issue (and this is "installation" section, not "migration"))
 
(65 intermediate revisions by 15 users not shown)
Line 1: Line 1:
 
[[Category:Kernel]]
 
[[Category:Kernel]]
{{Article summary start}}
+
[[ja:Modprobed-db]]
{{Article summary text|Describes the installation and usage of modprobe db.}}
+
{{Related articles start}}
{{Article summary heading|Related}}
+
{{Related|Kernels}}
{{Article summary wiki|Kernels}}
+
{{Related|Linux-ck}}
{{Article summary wiki|Linux-ck}}
+
{{Related articles end}}
{{Article summary end}}
+
  
{{AUR|modprobed_db}} keeps a running list of ALL modules ever probed on a system and allow for easy recall. This is very useful for users wishing to build a minimal kernel via a `make localmodconfig`which simply takes every module currently probed and switches everything BUT them off in the .config for a kernel resulting in smaller kernel packages and reduced compilation times.
+
{{AUR|modprobed-db}} keeps a running list of ALL modules ever probed on a system and allow for easy recall. This is very useful for users wishing to build a minimal kernel via a {{ic|make localmodconfig}} which simply takes every module currently probed and switches everything BUT them off in the {{ic|.config}} for a kernel resulting in smaller kernel packages and reduced compilation times.
  
 
== Installation and Setup ==
 
== Installation and Setup ==
#Download, build, and install {{AUR|modprobed_db}}.
+
The {{AUR|modprobed-db}} package is available from the [[AUR]].
#Edit {{ic|/etc/modprobed_db.conf}} and optionally add some modules to ignore to the IGNORE array. Some common ones are included by default.
+
 
 +
#Run {{ic|modprobed-db}} which will create {{ic|$XDG_CONFIG_HOME/modprobed-db.conf}} if one does not already exist.
 +
#Run {{ic|modprobed-db store}} to store the current loaded modules.
 +
 
 +
'''Optionally:''' add modules in the ignore array that you do NOT want counted, for example modules that get built or that are provided by another package.
 +
Some common ones are included by default:
 +
{{hc|$ cat ~/.config/modprobed-db.conf|<nowiki>
 +
IGNORE=(nvidia vboxdrv vboxnetflt vboxnetadp vboxpci lirc_dev lirc_i2c
 +
osscore oss_hdaudio oss_usb tp_smapi thinkpad_ec
 +
zavl znvpair zunicode zcommon zpios zfs spl splat)</nowiki>}}
  
 
== Usage ==
 
== Usage ==
=== Data Collection ===
+
Once the initial database has been created, simply use the system (insert USB sticks, use hardware that requires modules, mount filesystems that require modules, etc.) and periodically update the databases by one of two automatic methods:
The most convenient method to "use" modprobed_db is to simply add an entry in the root user's crontab to invoke {{ic|/usr/bin/modprobed_db store}} at some regular interval.
+
=== Cron ===
 +
The most convenient method to use modprobed-db is to simply add a crontab entry invoking {{ic|/usr/bin/modprobed-db store}} at some regular interval.
  
Example running the script once every 20 minutes:
+
Example running the script once every hour:
  # crontab -e
+
  $ crontab -e
  */20 * * * *  /usr/bin/modprobed_db store &> /dev/null
+
  0 */1 * * *  /usr/bin/modprobed-db store &> /dev/null
 +
 
 +
=== Systemd ===
 +
Systemd users not wishing to use cron may use the included user service: {{ic|modprobed-db.service}}. It will run modprobed-db in store mode once per hour, and at boot and on shutdown.
 +
 
 +
$ systemctl --user enable modprobed-db.service
 +
$ systemctl --user start modprobed-db.service
  
 
=== Data Recall ===
 
=== Data Recall ===
As mentioned earlier, this script is meant to be used in concert with the '''make localmodconfig''' step of compiling a kernel.  After the database has been adequately populated, simply invoke {{ic|/usr/bin/modprobed_db recall}} prior to compiling a kernel to load all modules followed by the '''make localmodconfig''' to do the magic.
+
As mentioned earlier, this script is meant to be used in concert with the '''make localmodconfig''' step of compiling a kernel.  After the database has been adequately populated, simply invoke {{ic|/usr/bin/modprobed-db recall}} prior to compiling a kernel to load all modules followed by the '''make localmodconfig''' to do the magic.
 +
 
 +
{{Note|Since {{ic|/usr/bin/modprobe}} requires root privileges, {{ic|/usr/bin/modprobed-db}} needs to be called as root or via sudo when users wish to recall the database.}}
  
==== Using the Official ARCH kernel PKGBUILD ====
+
==== Using the Official Arch kernel PKGBUILD ====
The official ARCH kernel's PKGBUILD does not have native support for this, but it is easily modified as follows:
+
The official Arch kernel's PKGBUILD does not have native support for this, but it is easily modified as follows:
  
 
{{bc| ...
 
{{bc| ...
Line 31: Line 48:
 
   make prepare
 
   make prepare
  
   /usr/bin/modprobed_db recall        <---- insert this line
+
   sudo /usr/bin/modprobed-db recall        <---- insert this line
   make localmodconfig             <---- insert this line
+
   make localmodconfig                       <---- insert this line
  
 
   # load configuration
 
   # load configuration
Line 44: Line 61:
  
 
==== Using Some Kernels in the AUR ====
 
==== Using Some Kernels in the AUR ====
Several kernel packages in the AUR have native support for modprobed_db in their PKGBUILD files.
+
Several kernel packages in the AUR have native support for modprobed-db in their PKGBUILD files.
 
For example:
 
For example:
*{{AUR|linux-bfs}}
+
*{{AUR|linux-bfs}}{{Broken package link|{{aur-mirror|linux-bfs}}}}
*{{AUR|linux-ck-systemtap}}
+
*{{AUR|linux-bridge-pl}}{{Broken package link|{{aur-mirror|linux-bridge-pl}}}}
*{{AUR|linux-ck-pax}}
+
*{{AUR|linux-ck-fbcondecor}}
 
*{{AUR|linux-ck}}
 
*{{AUR|linux-ck}}
*{{AUR|linux-ideapad}}
 
 
*{{AUR|linux-lqx}}
 
*{{AUR|linux-lqx}}
 +
*{{AUR|linux-lts310}}
 +
*{{AUR|linux-lts312}}
 +
*{{AUR|linux-lts-ck}}
 
*{{AUR|linux-pf}}
 
*{{AUR|linux-pf}}
*{{AUR|linux-rifs}}
+
*{{AUR|linux-uksm-ck}}{{Broken package link|{{aur-mirror|linux-uksm-ck}}}}
*{{AUR|linux-uksm-ck}}
+
*{{AUR|linux-ice}}
*{{AUR|linux-uksm}}
+
*{{AUR|linux-uksm}}{{Broken package link|{{aur-mirror|linux-uksm}}}}
  
 
Find which other packages use it:
 
Find which other packages use it:
 
  cd /scratch
 
  cd /scratch
  git clone http://pkgbuild.com/git/aur-mirror.git
+
  git clone --depth 1 http://pkgbuild.com/git/aur-mirror.git
  find /scratch/aur-mirror -iname "PKGBUILD" -print0 | xargs -0 grep "modprobed_db recall" | sort
+
  find /scratch/aur-mirror -iname "PKGBUILD" -print0 | xargs -0 grep -i 'modprobed-db recall\|modprobed_db recall' | sort
  
{{Note|The server pkgbuild.com is rather slow and the git clone can take a while.  You will be pulling down over 1.1 GB of data.}}
+
{{Note|The server pkgbuild.com is rather slow and the git clone can take a while.}}
 +
 
 +
Alternatively, download the .tar.xz snapshot (around 90 MB) by following [http://pkgbuild.com/git/aur-mirror.git/ this link] and then clicking on the latest commit message.
  
 
== Recommendations ==
 
== Recommendations ==
Line 73: Line 94:
 
*Users who plan to mount iso image file should do so (this will make sure to capture the '''loop''' and '''isofs''' modules).
 
*Users who plan to mount iso image file should do so (this will make sure to capture the '''loop''' and '''isofs''' modules).
 
*Users requiring encryption software such as {{Pkg|truecrypt}} should make sure to load it, and mount some encrypted containers to ensure that the needed crypto modules are in the db.
 
*Users requiring encryption software such as {{Pkg|truecrypt}} should make sure to load it, and mount some encrypted containers to ensure that the needed crypto modules are in the db.
 +
*Try-out different Linux-kernels; they may include modules not enabled in the default/other kernel(s)
  
 
== Suggested Modules ==
 
== Suggested Modules ==
Line 82: Line 104:
 
*isofs
 
*isofs
 
*loop
 
*loop
 +
*efivars
 
*vfat
 
*vfat
 +
*usb_storage
  
== Benefits of modprobed_db with '''make localmodconfig''' in custom kernels==
+
== Benefits of modprobed-db with '''make localmodconfig''' in custom kernels==
 
#Reduced kernel footprint on FS
 
#Reduced kernel footprint on FS
 
#Reduced compilation time
 
#Reduced compilation time
  
Comparisons using version 3.1.1-1 of the ARCH kernel (from ABS):
+
Comparisons using version 3.8.8-1 of the Arch kernel (from ABS):
  
{{Note| The /var/log/modprobed.db on the test machine contains 171 lines; YMMV based on specific usage and needs.}}
+
{{Note| The modprobed.db on the test machine contains 209 lines; YMMV based on specific usage and needs.}}
  
 
{| border="1"
 
{| border="1"
| '''Machine CPU''' || '''make localmodconfig'''||'''# of Modules''' || '''Modules' Size on HDD''' || '''Compilation Time'''
+
| '''Machine CPU''' || '''# of threads''' || '''make localmodconfig'''||'''# of Modules''' || '''Modules' Size on HDD''' || '''Compilation Time'''
 
|-
 
|-
| Intel C2Q @ 3.4 GHz || No || 3,148 || 46 MB || 13 min 50 sec
+
| Intel i7-3770K @ 4.50 GHz || 8 || No || 3,025 || 129 MB || 7 min 37 sec
 
|-  
 
|-  
| Intel C2Q @ 3.4 GHz || Yes || 596 || 13 MB || 5 min 30 sec
+
| Intel i7-3770K @ 4.50 GHz || 8 || Yes || 230 || 18 MB || 1 min 13 sec
 
|-  
 
|-  
| AMD Athlon XP 3200+ @ 2.2 GHz || No || 3,148 || 46 MB || 127 min 35 sec
+
| Intel Q9550 @ 3.40 GHz || 4 || No || 3,025 || 129 MB || 14 min 21 sec
 
|-  
 
|-  
| AMD Athlon XP 3200+ @ 2.2 GHz || Yes || 596 || 13 MB || 41 min 20 sec
+
| Intel Q9550 @ 3.40 GHz || 4 || Yes || 230 || 18 MB || 2 min 20 sec
 +
|-
 +
| Intel E5200 @ 3.33 GHz || 2 || No || 3,025 || 129 MB || 34 min 35 sec
 +
|-
 +
| Intel E5200 @ 3.33 GHz || 2 || Yes || 230 || 18 MB || 5 min 46 sec
 
|-  
 
|-  
 
|}
 
|}
  
*'''Net decrease in number of modules: 5.2x'''
+
*'''13x less modules built'''
*'''Net decrease in kernel footprint: 3.5x'''
+
*'''7x less space'''
*'''Net decrease in compilation time: 2.5x - 3.1x'''
+
*'''6x less compilation time'''
  
 
Number of modules found by:
 
Number of modules found by:
  find /lib/modules/3.1.1-1-ARCH/kernel -iname "*.ko.gz" | wc -l
+
  find /scratch/linux-3.8 -name '*.ko' | wc -l
  
 
Size on HDD found by:
 
Size on HDD found by:
  $ du -ch /lib/modules/3.1.1-1-ARCH/kernel | grep total
+
  find /scratch/linux-3.8 -name '*.ko' -print0 | xargs -0 du -ch
 +
 
 +
Compilation time found by entering a preconfigured linux-3.8.8 (using stock Arch config):
 +
$ time make -jx modules
  
Compilation time found by entering a preconfigured src/linux-3.1.1 (using stock ARCH config):
+
{{Note|The Arch standard is to gzip each module; the numbers shown in the table above are not gzip'ed but the savings ratio will be unaffected by this.}}
$ time make -jx bzImage modules
+

Latest revision as of 12:55, 23 April 2016

Related articles

modprobed-dbAUR keeps a running list of ALL modules ever probed on a system and allow for easy recall. This is very useful for users wishing to build a minimal kernel via a make localmodconfig which simply takes every module currently probed and switches everything BUT them off in the .config for a kernel resulting in smaller kernel packages and reduced compilation times.

Installation and Setup

The modprobed-dbAUR package is available from the AUR.

  1. Run modprobed-db which will create $XDG_CONFIG_HOME/modprobed-db.conf if one does not already exist.
  2. Run modprobed-db store to store the current loaded modules.

Optionally: add modules in the ignore array that you do NOT want counted, for example modules that get built or that are provided by another package. Some common ones are included by default:

$ cat ~/.config/modprobed-db.conf
IGNORE=(nvidia vboxdrv vboxnetflt vboxnetadp vboxpci lirc_dev lirc_i2c
osscore oss_hdaudio oss_usb tp_smapi thinkpad_ec
zavl znvpair zunicode zcommon zpios zfs spl splat)

Usage

Once the initial database has been created, simply use the system (insert USB sticks, use hardware that requires modules, mount filesystems that require modules, etc.) and periodically update the databases by one of two automatic methods:

Cron

The most convenient method to use modprobed-db is to simply add a crontab entry invoking /usr/bin/modprobed-db store at some regular interval.

Example running the script once every hour:

$ crontab -e
0 */1 * * *   /usr/bin/modprobed-db store &> /dev/null

Systemd

Systemd users not wishing to use cron may use the included user service: modprobed-db.service. It will run modprobed-db in store mode once per hour, and at boot and on shutdown.

$ systemctl --user enable modprobed-db.service
$ systemctl --user start modprobed-db.service

Data Recall

As mentioned earlier, this script is meant to be used in concert with the make localmodconfig step of compiling a kernel. After the database has been adequately populated, simply invoke /usr/bin/modprobed-db recall prior to compiling a kernel to load all modules followed by the make localmodconfig to do the magic.

Note: Since /usr/bin/modprobe requires root privileges, /usr/bin/modprobed-db needs to be called as root or via sudo when users wish to recall the database.

Using the Official Arch kernel PKGBUILD

The official Arch kernel's PKGBUILD does not have native support for this, but it is easily modified as follows:

 ...
  # get kernel version
  make prepare

  sudo /usr/bin/modprobed-db recall         <---- insert this line
  make localmodconfig                       <---- insert this line

  # load configuration
  # Configure the kernel. Replace the line below with one of your choice.
  #make menuconfig # CLI menu for configuration
  #make nconfig # new CLI menu for configuration
  #make xconfig # X-based configuration
  #make oldconfig # using old config from previous kernel version
  # ... or manually edit .config
 ...

Using Some Kernels in the AUR

Several kernel packages in the AUR have native support for modprobed-db in their PKGBUILD files. For example:

Find which other packages use it:

cd /scratch
git clone --depth 1 http://pkgbuild.com/git/aur-mirror.git
find /scratch/aur-mirror -iname "PKGBUILD" -print0 | xargs -0 grep -i 'modprobed-db recall\|modprobed_db recall' | sort
Note: The server pkgbuild.com is rather slow and the git clone can take a while.

Alternatively, download the .tar.xz snapshot (around 90 MB) by following this link and then clicking on the latest commit message.

Recommendations

It is recommended that users install the package and then "use" the system for a good amount of time to allow the database to grow based on usage and capture everything the system needs before building a kernel with a make localmodconfig. Some suggested actions to allow appropriate modules to load and get cataloged:

  • Insert every kind of removable media (USB, DVD, CD, etc.)
  • Use every device on the machine (wifi, network, USB stuff like cameras, ipods, etc.)
  • Mount every kind of filesystem one might typically use including ext2/3/4, fat, vfat, CIFS shares, NFS shares, etc.
  • Use as many applications (that one would normally use) as possible in order to capture modules on which they depend. For example, IP blocking/filtering software like pgl-cliAUR.
  • Users who plan to mount iso image file should do so (this will make sure to capture the loop and isofs modules).
  • Users requiring encryption software such as truecrypt should make sure to load it, and mount some encrypted containers to ensure that the needed crypto modules are in the db.
  • Try-out different Linux-kernels; they may include modules not enabled in the default/other kernel(s)

Suggested Modules

  • cifs
  • ext2
  • ext3
  • ext4
  • fat
  • isofs
  • loop
  • efivars
  • vfat
  • usb_storage

Benefits of modprobed-db with make localmodconfig in custom kernels

  1. Reduced kernel footprint on FS
  2. Reduced compilation time

Comparisons using version 3.8.8-1 of the Arch kernel (from ABS):

Note: The modprobed.db on the test machine contains 209 lines; YMMV based on specific usage and needs.
Machine CPU # of threads make localmodconfig # of Modules Modules' Size on HDD Compilation Time
Intel i7-3770K @ 4.50 GHz 8 No 3,025 129 MB 7 min 37 sec
Intel i7-3770K @ 4.50 GHz 8 Yes 230 18 MB 1 min 13 sec
Intel Q9550 @ 3.40 GHz 4 No 3,025 129 MB 14 min 21 sec
Intel Q9550 @ 3.40 GHz 4 Yes 230 18 MB 2 min 20 sec
Intel E5200 @ 3.33 GHz 2 No 3,025 129 MB 34 min 35 sec
Intel E5200 @ 3.33 GHz 2 Yes 230 18 MB 5 min 46 sec
  • 13x less modules built
  • 7x less space
  • 6x less compilation time

Number of modules found by:

find /scratch/linux-3.8 -name '*.ko' | wc -l

Size on HDD found by:

find /scratch/linux-3.8 -name '*.ko' -print0 | xargs -0 du -ch

Compilation time found by entering a preconfigured linux-3.8.8 (using stock Arch config):

$ time make -jx modules
Note: The Arch standard is to gzip each module; the numbers shown in the table above are not gzip'ed but the savings ratio will be unaffected by this.