Tuesday, March 8, 2016

How to create a custom kernel with BFQ on Fedora 23 with Nvidia binary drivers

BFQ is a block-layer I/O scheduler which greatly improves system responsiveness (up to 15 times) for users of HDD drives.

Since switching to Linux in 2008 the most annoying Linux issue to me has been ridiculously bad responsiveness of the OS when under I/O load, I've been wondering what exactly is the matter and these days I found out: apparently (most of) the blame is on CFQ - the current I/O scheduler on Linux.

I've been using BFQ for the past 2 days and had no issues and OS stuttering is pretty much gone.

Some distros ship with BFQ but Fedora (neither Ubuntu) doesn't ship with it, so here's the steps to create your own Linux kernel with BFQ enabled and used by default:

1. Install the devel packages to be able to compile the Linux kernel:

# dnf install kernel-devel openssl-devel

2. At the time of writing the latest version of BFQ was for the Linux 4.4 series, so grab the BFQ patches from here, you only need the 3 files starting with "000":
0001-block-cgroups-kconfig-build-bits-for-BFQ-v7r11-4.4.0.patch
0002-block-introduce-the-BFQ-v7r11-I-O-sched-for-4.4.0.patch
0003-block-bfq-add-Early-Queue-Merge-EQM-to-BFQ-v7r11-for.patch

3. Download a 4.4 Linux kernel, currently it's 4.4.4, from your favourite website, mine is lkml.org

4. Extract the files. From the command line can be done with:

$ tar xf linux-4.4.4.tar.xz

this will create a new folder "linux-4.4.4" with the Linux source code.

5. Put the 3 patch files from step two into this folder.

6. Applying the patches.
Open a terminal in this folder, then for each file call the patch command:
patch -p1 < 0001-block-cgroups-kconfig-build-bits-for-BFQ-v7r11-4.4.0.patch
patch -p1 < 0002-block-introduce-the-BFQ-v7r11-I-O-sched-for-4.4.0.patch
patch -p1 <  0003-block-bfq-add-Early-Queue-Merge-EQM-to-BFQ-v7r11-for.patch

In case you're wondering what "-p1" is for: each .patch file contains patches to multiple files and these are marked inside the patch file twice as the "old" and "new" file, respectively "a" and "b" so we need to strip these "a" and "b" file path prefixes otherwise the patch command won't find the correct files it needs to patch, and "p1" strips the first prefix of a file path which is what we want.
For example, this way the file path "a/block/Kconfig.iosched" becomes "block/Kconfig.iosched".
Respectively "p0" doesn't strip anything and "p2" strips two folder names, etc.

7. Although you just patched BFQ into your Linux source code, as of writing it's neither enabled nor chosen as the default I/O scheduler, so this has to be done manually from the linux "menuconfig":

$ make menuconfig

it'll take a few minutes to execute, then you'll be presented with a menu list with the following labels at the bottom "Select", "Exit", "Help", "Save" and "Load". From the menu list (with the up/down arrow keys) go to "Enable the block layer  --->", hit "Enter" on the keyboard, in the new menu list go to
"IO Schedulers --->", hit "Enter" again and here's the actual BFQ options.
The "Pause" key on the keyboard enables/disables an option (an option with a star - is enabled, without it - disabled). So:

7.1 Make sure that "BFQ I/O scheduler" has a star to the left (reminder - press the "Pause" key). BFQ is now enabled.

7.2 Now make sure it's the default I/O scheduler:
Go down to "Default I/O scheduler (...) --->", in the new menu list make sure BFQ has an "(X)" to the left, that is, that it's the default scheduler. After this, choose "Select" to go back to the menu list up one level where you can see at the bottom the "Save" label, choose it with the "Tab" key, hit "Enter", then "< Ok >", "< Exit >" and "Exit" again and again until you exit the Linux "menuconfig" completely.
BFQ is now both enabled and chosen by default.

8. Compile the Linux kernel.

$ make -j4

This starts the kernel compilation, if you have 8 cores use "make -j8" respectively. The compilation usually takes between 20 to 90 minutes depending on your computer performance.

9. Now the kernel has to be installed (as root!), first the so called modules:

# make modules_install

will take a few minutes to complete.

9.1 Copy your kernel to /boot:

# cp arch/x86/boot/bzImage /boot/vmlinuz-4.4.4

9.2 Create initrd:

# mkinitrd --force /boot/initramfs-4.4.4.img 4.4.4

10. Regenerate Grub config to add the new kernel to the boot list:

EFI motherboards:
# grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

BIOS motherboards:
# grub2-mkconfig -o /boot/grub2/grub.cfg

11. The kernel 4.4.4 is now fully installed but not registered with dkms. The Nvidia binary driver relies on dkms to automatically recompile its driver modules for any new kernel, so let's register our new kernel before rebooting the OS, otherwise after we reboot we'll get a broken desktop.

# dkms autoinstall -k 4.4.4

12. Reboot
You should now be running Linux with the BFQ scheduler used by default, to check it's true, from the command line do this:

$ cat /sys/block/sda/queue/scheduler

It should print "noop deadline cfq [bfq]", the fact that "[bfq]" is there and it's in square brackets means it's enabled and used by default for your HDD (sda).

Congratulations!
Your OS stuttering when under I/O should now be a thing of the past.

More info about BFQ at:
http://algo.ing.unimo.it/people/paolo/disk_sched/
I'm not the author of BFQ, just a happy user.






1 comment:

  1. To save life, please use Archlinux, and simply run 'yaourt -S nvidia-ck'.

    ReplyDelete