SSD caching using Linux and bcache

A couple of manufacturers are selling solutions to speed up your big HDD using a relative small SSD. There are techniques like:

But there is also a lot of development on the Linux front. There is Flashcache developed by Facebook, dm-cache by VISA and bcache by Kent Overstreet. The last one is interesting, because it’s a patch on top of the Linux kernel and will hopefully be accepted upstream some day.

Hardware setup

In my low power homeserver I use a 2TB Western Digital Green disk (5400 RPM). To give bcache a try I bought a 60GB Intel 330 SSD. Some facts about these data-carriers. The 2TB WD can do about 110 MB/s of sequential reads/writes. This traditional HDD does about 100 random operations per second. The 60GB Intel 330 can sequentially read about 500 MB/s and write about 450 MB/s. Random reads are done in about 42.000 operations per second, random writes in about 52.000. The SSD is much faster!

The image below shows the idea of SSD caching. Frequently accessed data is cached on the SSD to gain better read performance. Writes can be cached on the SSD using the writeback mechanism.

Prepare Linux kernel and userspace software

To be able to use bcache, there are 2 things needed:

  1. A bcache patched kernel
  2. bcache-tools for commands like make-bcache and probe-bcache

I used the latest available 3.2 kernel. The bcache-3.2 branch from Kent’s git repo merged successfully. Don’t forget to enable the BCACHE module before compiling.

On my low power home server I use Debian. Since there is no bcache-tools Debian package available yet, I created my own. Fortunately damoxc already packaged bcache-tools for Ubuntu once.

Debian package: pommi.nethuis.nl/…/bcache/
Git web: http://git.nethuis.nl/?p=bcache-tools.git;a=summary
Git: http://git.nethuis.nl/pub/bcache-tools.git

Bcache setup

Unfortunately bcache isn’t plug-and-play. You can’t use bcache with an existing formatted partition. First you have to create a caching device (SSD) and a backing device (HDD) on top of two existing devices. Those devices can be attached to each other to create a /dev/bcache0 device. This device can be formatted with your favourite filesystem. The creation of a caching and backing device is necessary because it’s a software implementation. Bcache needs to know what is going on. For example when booting, bcache needs to know what devices to attach to each other. The commands for this procedure are shown in the image below.

After this I had a working SSD caching setup. Frequently used data is stored on the SSD. Accessing and reading frequently used files is much faster now. By default bcache uses writethrough caching, which means that only reads are cached. Writes are being written directly to the backing device (HDD).

To speed up the writes you have to enable writeback caching. But you have to take in mind, there is a risk of losing data when using a writeback cache. For example when there is a power failure or when the SSD dies. Bcache uses a fairly simple journalling mechanism on the caching device. In case of a power failure bcache will try to recover the data. But there is a chance you will end up with corruption.

echo writeback > /sys/block/sda/sda[X]/bcache/cache_mode

When writes are cached on the caching device, the cache is called dirty. The cache is clean again, when all cached writes have been written to the backing device. You can check the state of the writeback cache via:

cat /sys/block/sda/sda[X]/bcache/state

To detach the caching device from the backing device run the command below (/dev/bcache0 will still be available). This can take a while when the write cache contains dirty data, because it must be written to the backing device first.

echo 1 > /sys/block/sda/sda[X]/bcache/detach

Attach the caching device again (or attach another caching device):

echo [SSD bcache UUID] > /sys/block/sda/sda[X]/bcache/attach

Unregister the caching device (can be done with or without detaching) (/dev/bcache0 will still be available because of the backing device):

echo 1 > /sys/fs/bcache/[SSD bcache UUID]/unregister

Register the caching device again (or register another caching device):

echo /dev/sdb[Y] > /sys/fs/bcache/register

Attach the caching device:

echo [SSD bcache UUID] > /sys/block/sda/sda[X]/bcache/attach

Stop the backing device (after unmounting /dev/bcache0 it will be stopped and removed, don’t forget to unregister the caching device):

echo 1 > /sys/block/sda/sda[X]/bcache/stop

Benchmark

To benchmark this setup I used two different tools. Bonnie++ and fio – Flexible IO tester.

 

Bonnie++

Unfortunately Bonnie++ isn’t that well suited to test SSD caching setups.

This graph shows that I’m hitting the limit on sequential input and output in the HDD-only and SSD-only tests. The bcache test doesn’t show much difference to the HDD-only test in this case. Bonnie++ isn’t able to warm up the cache and all sequential writes are bypassing the write cache.

In the File metadata tests the performance improves when using bcache.

 

fio

The Flexible IO tester is much better to benchmark these situations. For these tests I used the ssd-test example jobfile and modified the size parameter to 8G.

HDD-only:
seq-read: io=4084MB, bw=69695KB/s, iops=17423, runt= 60001msec
rand-read: io=30308KB, bw=517032B/s, iops=126, runt= 60026msec
seq-write: io=2792MB, bw=47642KB/s, iops=11910, runt= 60001msec
rand-write: io=37436KB, bw=633522B/s, iops=154, runt= 60510msec

SSD-only:
seq-read: io=6509MB, bw=110995KB/s, iops=27748, runt= 60049msec
rand-read: io=1896MB, bw=32356KB/s, iops=8088, runt= 60001msec
seq-write: io=2111MB, bw=36031KB/s, iops=9007, runt= 60001msec
rand-write: io=1212MB, bw=20681KB/s, iops=5170, runt= 60001msec

bcache:
seq-read: io=4127.9MB, bw=70447KB/s, iops=17611, runt= 60001msec
rand-read: io=262396KB, bw=4367.8KB/s, iops=1091, runt= 60076msec
seq-write: io=2516.2MB, bw=42956KB/s, iops=10738, runt= 60001msec
rand-write: io=2273.4MB, bw=38798KB/s, iops=9699, runt= 60001msec

In these tests the SSD is much faster with random operations. With the use of bcache random operations are done a lot faster in comparison to the HDD-only tests. It’s interesting that I’m not able to hit the sequential IO limits of the HDD and SSD in these tests. I think this is because my CPU (Intel G620) isn’t powerful enough for these tests. fio hits the IO limits of the SSD in an another machine with a Intel i5 processor.

25 thoughts on “SSD caching using Linux and bcache

  1. ascii

    Hi Pommi,

    did you succeeed in getting a clean bcache patch to apply against vanilla kernel? I would like to follow the your and damoxc in order to create a complete howto on how to get bcache up and running on ubuntu Precise.

    Thanks,
    ascii

  2. Morgan S

    This was a very well written article and was a lot easier to understand than the documentation I had first perused when starting to use bcache. In your graphic, you show device registration with cat although I believe echo is necessary instead. Cheers!

  3. Zoblu

    Do you know if it’s possible to restart a stopped¬†cache?
    I’m trying to reload bcache after extending the bdev, and I can’t rmmod or reboot.

  4. pommi Post author

    Hi Zoblu, I was able to reuse my caching device by stopping it (see my last commands above) and attaching it again to a bcache backing device. Nothing special. One thing I have to say is that I don’t use bcache as a dynamic kernel module, but static. So in my case I never rmmod something.

  5. Reg

    Hi, thanks for this.
    I tried the deb package, but got an alarm saying “wrong architecture”.
    Do you hae an x86 version ? Maybe that should be x64, anyway I have an intel i7.

  6. pommi Post author

    Hi Reg, the package I created is for 64bit architecture. But my package is already 1 year old. Probably it doesn’t work with later versions of bcache. I suggest to keep watching this thread for an official Debian package.

  7. Reg

    Thans for the response and the link to the thread, I will keep a look-out there.

    I suspect that the wrong architecture message may have arisen from not finding a suitable file system in sys/fs/
    This is only a GUESS, I am a complete novice on this (-:

  8. Mez

    Hi Pommi!

    do you know any way to check if the bcache is really working? For example, check whether things are being written to the ssd and are being read out?

    Thenks

  9. pommi Post author

    Check the files in /sys/block/bcache0/bcache/stats_total like:
    tail /sys/block/bcache0/bcache/stats_total/*

  10. yixuan

    Hello,
    Can you help me know your test config for fio? I am testing ssd as cache in my environment, when I used iometer, it will show similar result as HDD only.
    I want to use fio or other tools to confirm.
    Thanks,
    Eugene

  11. pommi Post author

    From the blogpost above: “For these tests I used the ssd-test example jobfile and modified the size parameter to 8G.

    This is the ssd-test example jobfile.

  12. evan

    Hi pommi,

    I’m newbie of SSD caching, and I’m going to install my new data server with SSD caching technology.
    However, what will happen to my server, if the SSD crash for any reason.
    Will it also crush in the moment that SSD failed, or just no caching capability ?

    This article is very useful.
    Thanks for you sharing.

    evan

  13. pommi Post author

    It depends. If you enable writeback caching there is a big chance your filesystem will be corrupt. When you use it in writethrough mode you won’t lose data. I don’t know exactly if Linux crashes in case of an SSD failure. I see a lot of people using bcache on top 2 SSD’s in RAID1. Maybe this is interesting to read: http://www.spinics.net/lists/linux-bcache/msg01461.html

  14. Ricky

    Hi pommi,

    Will bcache wipe out the existing data in backing device when creating them?

    Does bcache improve overall I/O when using application / boot?

  15. pommi Post author

    Hi Ricky, bcache will indeed wipe the existing data, but there is a tool to convert an existing formatted partition to a bcache backing device. Please search through the bcache mailinglist for answers.

  16. Glenn

    Hi pommi,

    First I have to say thanks for an awesome guide.
    I have a few questions though, which I hope you are up for answering.

    I’m running Debian squeeze, with a home brew 3.12 kernel bcache enabled of course. When I reboot the machine, the bcache drive is not showed in /dev/ – Do you have any pointers, this happens both with and without writeback enabled.

  17. pommi Post author

    Hi Glenn, I’m currently using bcache in with an encrypted backing device. Therefor I’m using a custom bash script to mount my bcache devices so I can enter the passphrase of my encrypted drive too. What I’m currently doing to recreate the bcache device after a reboot is:

    echo /dev/mapper/data > /sys/fs/bcache/register_quiet
    echo /dev/sdb2 > /sys/fs/bcache/register_quiet

    BUT when you have bcache-tools installed there is a udev rule that will do that for you. bcache-tools isn’t in Debian yet, but they are working on it.

  18. Glenn

    Hi Pommi,

    I am using bcache-tools in fact the version you’ve posted, all is fine except after boot. If I register the drives after boot they work just fine:
    echo /dev/md3 > /sys/fs/bcache/register — backing device
    echo /dev/md2 > /sys/fs/bcache/register — caching device

    And then mount them. It all plays nicely, but I would prefer it done automatic.

  19. pommi Post author

    I’m sorry to say, but please don’t use my 0.0.2-1 packaged version of bcache-tools anymore. It’s old and probably isn’t compatible with the latest bcache versions.

    I’ve just compiled a new bcache-tools Debian package based on 1.0.4-1 from mentors.debian.org on a Debian Wheezy machine. Use at your own risk.

  20. Glenn

    Tested and not working on a Debian squeeze installation:
    (Reading database ... 99388 files and directories currently installed.)
    Preparing to replace bcache-tools 0.0.2-1 (using bcache-tools_1.0.4-1_amd64.deb) ...
    dpkg (subprocess): unable to execute new pre-installation script (/var/lib/dpkg/tmp.ci/preinst): Exec format error
    dpkg: error processing bcache-tools_1.0.4-1_amd64.deb (--install):
    subprocess new pre-installation script returned error exit status 2
    dpkg (subprocess): unable to execute new post-removal script (/var/lib/dpkg/tmp.ci/postrm): Exec format error
    dpkg: error while cleaning up:
    subprocess new post-removal script returned error exit status 2
    Errors were encountered while processing:
    bcache-tools_1.0.4-1_amd64.deb

  21. pommi Post author

    Hmm, yeah, this is the “Use at your own risk” part. Works fine on Wheezy. Squeeze will be unsupported in ~4 months, so I would say: Upgrade to Wheezy ;-)

  22. Glenn

    Tested and installed successfully.

    Did a restart and mounts came up correctly – Your patch is working excellent thanks a lot!
    No more need for custom script with this.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>