Bcache

From ArchWiki
Revision as of 21:02, 5 March 2013 by Bobpaul (Talk | contribs) (Troubleshooting)

Jump to: navigation, search

Introduction

Bcache allows one to use an SSD as a read/write cache (in writeback mode) or read cache (writethrough or writearound) for another blockdevice (generally a rotating HDD or array. This article will show how to install arch using Bcache as the root partition. For an intro to bcache itself, see the bcache homepage. Be sure to read and reference the bcache manual. Bcache is currently in the process of getting merged into the kernel.

An alternative to Bcache is Facebook's Flashcache

Warning: Until bcache patches are accepted into mainline, installation is non-trivial. Follow the instructions carefully and read the upstream bcache documentation. Commands are included as examples only, and may differ slightly on your setup.
Warning: Bcache requires the backing device is formatted as a bcache block device. This is a destructive processes, so be sure you back up any important data first.

Temporary install to SSD

The ArchLinux install disk does not include bcache in its kernel. You could either re-master to the install disk to include this kernel, or do the install twice. The example below is for installing twice.

  1. Install archlinux as normal to the SSD. Don't go crazy with partitions, just a / and an EFI system partition if you're not booting bios mode. This is just the temporary system we'll use to install the final system.
  2. Reboot to verify the install in step 1 worked.
  3. Build and install the bcache enabled kernel and tools (see below). Be sure you install both the kernel and the kernel headers packages that are created.
  4. Reboot to the bcache enabled kernel

Final install to the HDD

1. Partition your hdd
grub can't handle bcache, so you'll need at least 2 partitions (boot and one for the bcache backing device). If you're doing UEFI, you'll need an EFI System Partition (ESP) as well.

  ex:
     1            2048           22527   10.0 MiB    EF00  EFI System
     2           22528          432127   200.0 MiB   8300  arch_boot
     3          432128       625142414   297.9 GiB   8300  bcache_backing

2. Configure your HDD as a bcache backing device.

 # make-bcache -B /dev/sda3
Note: GRUB2 either needs an EFI partition, "bios boot" partition, or an 'embedding area' of 2KB or so after the MBR that doesn't get overwritten. Also, since GRUB2 does not know about bcache, so you also need a /boot partition that grub can read. Partitioning /dev/sda is a must unless you're booting from a different device.

3. Register any bcache devices with the kernel (this needs to done every bootup)

  for i in /dev/sd*; do echo $i; echo $i > /sys/fs/bcache/register_quiet; done

You now have a /dev/bcache0 device

Note: the bcache user manual says to do "echo /dev/sd* > /sys/fs/bcache/register_quiet", but this didn't work for me

7. Format the bcache device. Use LLVM or btrfs subvolumes if you want to divide up the /dev/bcache0 device how you like (ex for seperate /, /home, /var, etc).

  # mkfs.btrfs /dev/bcache0
  # mount /dev/bcache0 /mnt/
  # btrfs subvolume create /mnt/root
  # btrfs subvolume create /mnt/home
  # umount /mnt

8. Reinstall to the bcache volume (alternatively you can rsync the install you made on the SSD)

  # mkfs.ext4 /dev/sda2
  # mkfs.msdos /dev/sda1 (if your ESP is at least 500MB, use mkfs.vfat to make a FAT32 partition instead)
  # pacman -S arch-install-scripts
  # mount /dev/bcache0 -o subvol=root,compress=lzo /mnt/
  # mkdir /mnt/boot
  # mkdir /mnt/home
  # mount /dev/bcache0 -o subvol=home,compress=lzo /mnt/home
  # mount /dev/sda2 /mnt/boot
  # mkdir /boot/efi
  # mount /dev/sda1 /mnt/boot/efi/

do the rest of the installation as normal

9. Build and install the bcache enabled kernel and tools (see below). Feel free to rsync the packages you already built on the SSD to avoid compiling again.

10. Edit the initramfs to register bcache devices on bootup (or it won't find root) Basically we need to do the same thing as in step 3, but every time we boot before the /dev/bcache0 device is available for root. This is done with Mkinitcpio

  • a. make a file /usr/lib/initcpio/install/bcache which contains
   #!/bin/bash
   
   build() {
       add_runscript
   }
   
   help() {
       cat <<HELPEOF
   This hook supports early, automatic registration of bcache devices. This is necessary
   if your root filesystem is on a /dev/bcacheX device
   HELPEOF
   }
  • b. make a file /usr/lib/initcpio/hooks/bcache which contains
   #!/usr/bin/ash
   run_hook() {
       for i in /dev/sd*; do
           echo $i > /sys/fs/bcache/register_quiet
       done
   }
  • c. in /etc/mkinitcpio.conf, add bcache to the HOOKS array. ORDER MATTERS. It needs to be after udev and block and before filesystems.
  • d. # mkinitcpio -p linux-bcache

11. Reboot and make sure it works from the /dev/bcache0 device

12. delete the partitions from /dev/sdb, format it as a caching device, and link it to the backing device

   # parted /dev/sdb mklabel msdos; dd if=/dev/zero of=/dev/sdb bs=512 count=1
   # make-bcache -C /dev/sdb
   # echo <Set UUID from previous command> > /sys/block/bcache0/bcache/attach
   # echo /dev/sdb > /sys/fs/bcache/register

Troubleshooting

If on step 11 you're sent to a busybox shell during root with an error about unable to find root device, you'll see (with ls) that /dev/bcache0 doesn't exist. This probably means something didn't work in step 10. In the mean time, manually register the bcache devices with the kernel as we did in step 3, ensure the /dev/bcache0 device node exists, then type "exit" to resume booting.

Kernel Building

Create the patch file

The code we're interested in is stored in the bcache_for_mainline branch of the bcache git repository.

1. Clone the bcache repo and checkout the bcache-for-upstream branch:

  $ git clone http://evilpiepirate.org/git/linux-bcache.git
  $ cd linux-bcache
  $ git checkout bcache-for-upstream
  Branch bcache-for-upstream set up to track remote branch bcache-for-upstream from origin.
  Switched to a new branch 'bcache-for-upstream'


2. Use "git log" to find the last time bcache was merged with Linus's tree

  $ git log

You should see a bunch of commits by Kent Overstreet. Scroll down an eventually you'll find one by Linus Torvalds. Copy the commit hash number for Linus's commit. Ex:

  commit 6e0298acc81e98cf1516358f62d6f43187b0bfa3
  Author: Kent Overstreet <koverstreet@google.com>
  Date:   Wed Sep 21 21:43:05 2011 -0700
   
      Revert "rw_semaphore: remove up/down_read_non_owner"
       
       This reverts commit 11b80f459adaf91a712f95e7734a17655a36bf30.
       
       Bcache needs rw semaphores for cache coherency in writeback mode -
       writes have to take a read lock on a per cache device rw sem, and
       release it when the bio completes.
       
       But since this is for bios it's naturally not in the context of the
       process that originally took the lock.
       
       Signed-off-by: Kent Overstreet <koverstreet@google.com>
       CC: Christoph Hellwig <hch@infradead.org>
       CC: David Howells <dhowells@redhat.com>
   
   commit 9931faca02c604c22335f5a935a501bb2ace6e20
   Author: Linus Torvalds <torvalds@linux-foundation.org>
   Date:   Wed Jan 9 18:59:55 2013 -0800
   
       Linux 3.8-rc3

3. Create a patch for all the stuff since the last merge with Linus.

   $ git format-patch 9931faca02c --stdout > bcache-patch-3.8-rc3.patch
Note: You don't need the entire commit hash, just enough for it to be unique. 6-8chars is generally acceptable. Notice I named the patch file based on Linus's commit message.

Build the kernel

The patch made above should apply to a recent kernel. Thankfully, ArchLinux uses a very up to date kernel. It's possible you might have issues with the patch not applying cleanly. That's beyond the scope of this documentation.

Follow the instructions on Kernels/Compilation/Arch_Build_System to build a kernel with the patch applied. Below are the changes I had to make

1. Copy the bcache-patch-3.8-rc3.patch we made earlier into the kernel folder you created in "Getting the Ingredients".

2. Generate an md5sum for the patch using "md5sum bcache-patch-3.8-rc3.patch"

2. Edit the PKGBUILD file and

  a. change pkgbase to "linux-bcache".
  b. add "bcache-patch-3.8-rc3.patch" to the end of the source list.
  c. add the md5sum from the previous step to the to the md5sums list
  d. uncomment "Enable make menuconfig" near the end of the build() function
  e Near the top of the build() function, add the lines to apply the patch
   
 # bcache patch against 3.8-rc3
 # http://atlas.evilpiepirate.org/git/linux-bcache.git/log/?h=bcache-for-upstream
 patch -Np1 -i "${srcdir}/bcache-patch-3.8-rc3.patch"


3. Run makepkg. When the kernel config pops up, be sure to enable bcache. Unless you plan to debug, there's no need for the bcache debugging options.

  Block device as cache (BCACHE) [N/m/y/?] (NEW) y
   Bcache debugging (BCACHE_DEBUG) [N/y/?] (NEW) n
   Extended runtime checks (BCACHE_EDEBUG) [N/y/?] (NEW) n
   Debug closures (BCACHE_CLOSURES_DEBUG) [N/y/?] (NEW) n
   Cgroup controls for bcache (CGROUP_BCACHE) [N/y/?] (NEW) n
Note: If you choose module, you'll need to be sure to include the module in your initrd. If you're not prompted for these questions, you can find them when "make menuconfig" opens.

4. Install the kernel packages. makepkg will have made a kernel package, docs package, and a headers package.