linux · kernel · vm

Minimizing Linux Kernel Size

WHY?

I’ve been using Digital Ocean for a few years and always had the cheapest $5 droplet that runs 32bit Linux. 64bit Linux can run as well but 512MB is not enough to run everything I want. I had 13.10 32bit Ubuntu running since forever and finally had the time to upgrade it to 16.04. Right after the upgrade, I noticed that new system had 505MB total memory while the old one had 508MB. Also, new system had less user space memory available because it had new daemons running that were not running on 13.10.

I want to squeeze every spare CPU cycle out of this box and every spare Megabyte, just because I can, so let’s see what can be done.

# 16.04.1
uname -a
Linux egreex 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:46:51 UTC 2016 i686 i686 i686 GNU/Linux
cat /proc/meminfo |grep MemTotal
MemTotal:         505467 kB
# 13.10
uname -a
Linux egreex 3.8.0-29-generic #42-Ubuntu SMP Tue Aug 13 23:12:18 UTC 2013 i686 i686 i686 GNU/Linux
cat /proc/meminfo |grep MemTotal
MemTotal:         508036 kB

On a system with only 512MB, every Megabyte counts. Let’s do some cleaning up and I’ll start with the user land. Check the process list. Snapd took 12MB of RSS RAM. WHAAT? KILL and apt-get remove! Can’t afford it to run on my tiny droplet. Remove other junk not suitable for this box as well:

apt-get remove snapd open-iscsi liblvm2cmd2.02:i386 dmeventd:i386 liblvm2app2.2:i386 lvm2

I noticed there were some modules in lsmod list and I tried to unload modules like btrfs xfs raid1 raid0 floppy parport_pc raid10 raid456 etc which are loaded on boot, but not used. It did not help, memory usage did not change =((. One trick to get a bit more memory, is to create a swap. We are running on SSD so it should work fast and make our box a bit happier:

dd if=/dev/zero of=/swap bs=1M count=1000
mkswap /swap

# add swap to fstab
swapon -a

# add some memory tricks to sysctl
# will be able to commit swap+0.75*RAM
echo "
vm.overcommit_memory=2
vm.overcommit_ratio=75
vm.min_free_kbytes=8192
" >> /etc/sysctl.conf

Kernel Time

Now it is time to do some kernel magic. Can you compile your own kernel on a tiny 512MB droplet? Turns out, yes you can:

wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.4.60.tar.xz
tar xaf linux-4.4.60.tar.xz
apt-get install build-essential dpkg-dev build-dep libncurses5-dev
make mrproper
make menuconfig
# add your own prefix, like -do1 and recompile
... very long output skipped ...
  IHEX    firmware/cpia2/stv0672_vp4.bin
  IHEX    firmware/yam/1200.bin
  IHEX    firmware/yam/9600.bin

real    148m56.716s
user    131m45.256s
sys     11m23.888s

Took more than 2 hours to build, but compiled successfully. I started removing bits and pieces from the .config file using make menuconfig, took me quite a few attempts to come up with a new kernel config. Here is what I changed:

  • enabled optimize for size
  • ramdisk compression, leave gzip only
  • partition types remove all except needed
  • io schedulers leave noop (we are on ssd raid)
  • disable SMP (we have 1 cpu)
  • disable stuff like AMD chipsets etc
  • disable xen
  • disable kvm debug
  • change the cpu to Core2/Newer XEON
  • supported vendors -> leave Intel only
  • preemtion -> server
  • high memory support -> off
  • disable pae, ksm, thp, eisa
  • disable power management (we are on a VM!!) , but leave acpi and cpu freq scaling
  • disable wireless and amateur devices,wimax, rf switch, plan9, caif, nfc
  • in networking options, disable openvswitch, appletalk, ipx, decnet,tipc, mpls
  • device drivers: disable parallel port, block devices -> floppy
  • scsi low level drivers - almost everything should be modularized, remove real devices
  • sata - removed all, there is no sata, we have virtio
  • remove raid support
  • disable mac drivers
  • network device support - remove all wireless, wan, isdn, ethernet, arcnet, fddi, all other devices not present on the system (but make sure you have virtio_net !!)
  • remove all multimedia, firewire and sound cards, leave only intel video cards, x86 platform specific
  • remove all exotic filesystems support
  • disable selinux
  • make sure you have apparmor enabled
  • make sure all firewall functions to remain in the kernel - you might need it
  • do not touch cgroups/namespaces - we are running on systemd
  • a lot more, have a look and save some RAM!!!

Let’s compare size. We had 505MB total RAM with the stock kernel and kernel size was 6.8MB:

-rw------- 1 root root 6826496 Mar 31 16:42 vmlinuz-4.4.0-72-generic

Custom kernel:

uname -a
Linux egreex 4.4.60-do1 #1 Mon Apr 10 12:56:33 UTC 2017 i686 i686 i686 GNU/Linux

ls -l /boot/vmlinuz-4.4.60-do1
-rw-r--r-- 1 root root 4024736 Apr 10 13:27 vmlinuz-4.4.60-do1

# memtotal
MemTotal:         510908 kB

That is almost 511MB total RAM, which is 5.4MB more than with stock kernel. We’re also optimized to run on a Core2/XEON which is what DO offer:

cat /proc/cpuinfo | grep 'model'
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz

Not a huge improvement but it comes for free, just recompile the kernel. The kernel .config is here and you should compile your kernel, too!

Published:
comments powered by Disqus