1 Say you've got a big slow raid 6, and an ssd or three. Wouldn't it be
2 nice if you could use them as cache... Hence bcache.
4 Wiki and git repositories are at:
5 http://bcache.evilpiepirate.org
6 http://evilpiepirate.org/git/linux-bcache.git
7 http://evilpiepirate.org/git/bcache-tools.git
9 It's designed around the performance characteristics of SSDs - it only allocates
10 in erase block sized buckets, and it uses a hybrid btree/log to track cached
11 extents (which can be anywhere from a single sector to the bucket size). It's
12 designed to avoid random writes at all costs; it fills up an erase block
13 sequentially, then issues a discard before reusing it.
15 Both writethrough and writeback caching are supported. Writeback defaults to
16 off, but can be switched on and off arbitrarily at runtime. Bcache goes to
17 great lengths to protect your data - it reliably handles unclean shutdown. (It
18 doesn't even have a notion of a clean shutdown; bcache simply doesn't return
19 writes as completed until they're on stable storage).
21 Writeback caching can use most of the cache for buffering writes - writing
22 dirty data to the backing device is always done sequentially, scanning from the
23 start to the end of the index.
25 Since random IO is what SSDs excel at, there generally won't be much benefit
26 to caching large sequential IO. Bcache detects sequential IO and skips it;
27 it also keeps a rolling average of the IO sizes per task, and as long as the
28 average is above the cutoff it will skip all IO from that task - instead of
29 caching the first 512k after every seek. Backups and large file copies should
30 thus entirely bypass the cache.
32 In the event of a data IO error on the flash it will try to recover by reading
33 from disk or invalidating cache entries. For unrecoverable errors (meta data
34 or dirty data), caching is automatically disabled; if dirty data was present
35 in the cache it first disables writeback caching and waits for all dirty data
39 You'll need make-bcache from the bcache-tools repository. Both the cache device
40 and backing device must be formatted before use.
41 make-bcache -B /dev/sdb
42 make-bcache -C /dev/sdc
44 make-bcache has the ability to format multiple devices at the same time - if
45 you format your backing devices and cache device at the same time, you won't
46 have to manually attach:
47 make-bcache -B /dev/sda /dev/sdb -C /dev/sdc
49 bcache-tools now ships udev rules, and bcache devices are known to the kernel
50 immediately. Without udev, you can manually register devices like this:
52 echo /dev/sdb > /sys/fs/bcache/register
53 echo /dev/sdc > /sys/fs/bcache/register
55 Registering the backing device makes the bcache device show up in /dev; you can
56 now format it and use it as normal. But the first time using a new bcache
57 device, it'll be running in passthrough mode until you attach it to a cache.
58 If you are thinking about using bcache later, it is recommended to setup all your
59 slow devices as bcache backing devices without a cache, and you can choose to add
60 a caching device later.
61 See 'ATTACHING' section below.
63 The devices show up as:
67 As well as (with udev):
69 /dev/bcache/by-uuid/<uuid>
70 /dev/bcache/by-label/<label>
74 mkfs.ext4 /dev/bcache0
75 mount /dev/bcache0 /mnt
77 You can control bcache devices through sysfs at /sys/block/bcache<N>/bcache .
78 You can also control them through /sys/fs//bcache/<cset-uuid>/ .
80 Cache devices are managed as sets; multiple caches per set isn't supported yet
81 but will allow for mirroring of metadata and dirty data in the future. Your new
82 cache set shows up as /sys/fs/bcache/<UUID>
87 After your cache device and backing device are registered, the backing device
88 must be attached to your cache set to enable caching. Attaching a backing
89 device to a cache set is done thusly, with the UUID of the cache set in
92 echo <CSET-UUID> > /sys/block/bcache0/bcache/attach
94 This only has to be done once. The next time you reboot, just reregister all
95 your bcache devices. If a backing device has data in a cache somewhere, the
96 /dev/bcache<N> device won't be created until the cache shows up - particularly
97 important if you have writeback caching turned on.
99 If you're booting up and your cache device is gone and never coming back, you
100 can force run the backing device:
102 echo 1 > /sys/block/sdb/bcache/running
104 (You need to use /sys/block/sdb (or whatever your backing device is called), not
105 /sys/block/bcache0, because bcache0 doesn't exist yet. If you're using a
106 partition, the bcache directory would be at /sys/block/sdb/sdb2/bcache)
108 The backing device will still use that cache set if it shows up in the future,
109 but all the cached data will be invalidated. If there was dirty data in the
110 cache, don't expect the filesystem to be recoverable - you will have massive
111 filesystem corruption, though ext4's fsck does work miracles.
116 Bcache tries to transparently handle IO errors to/from the cache device without
117 affecting normal operation; if it sees too many errors (the threshold is
118 configurable, and defaults to 0) it shuts down the cache device and switches all
119 the backing devices to passthrough mode.
121 - For reads from the cache, if they error we just retry the read from the
124 - For writethrough writes, if the write to the cache errors we just switch to
125 invalidating the data at that lba in the cache (i.e. the same thing we do for
126 a write that bypasses the cache)
128 - For writeback writes, we currently pass that error back up to the
129 filesystem/userspace. This could be improved - we could retry it as a write
130 that skips the cache so we don't have to error the write.
132 - When we detach, we first try to flush any dirty data (if we were running in
133 writeback mode). It currently doesn't do anything intelligent if it fails to
134 read some of the dirty data, though.
140 A) Your bcache doesn't start.
141 Starting and starting a bcache with a missing caching device
143 Registering the backing device doesn't help, it's already there, you just need
144 to force it to run without the cache:
145 host:~# echo /dev/sdb1 > /sys/fs/bcache/register
146 [ 119.844831] bcache: register_bcache() error opening /dev/sdb1: device already registered
148 Next, you try to register your caching device if it's present. However if it's
149 absent, or registration fails for some reason, you can still start your bcache
150 without its cache, like so:
151 host:/sys/block/sdb/sdb1/bcache# echo 1 > running
154 B) Bcache not finding its cache and not starting
157 host:/sys/block/md5/bcache# echo 0226553a-37cf-41d5-b3ce-8b1e944543a8 > attach
158 [ 1933.455082] bcache: bch_cached_dev_attach() Couldn't find uuid for md5 in set
159 [ 1933.478179] bcache: __cached_dev_store() Can't attach 0226553a-37cf-41d5-b3ce-8b1e944543a8
160 [ 1933.478179] : cache set not found
162 In this case, the caching device was simply not registered at boot or
163 disappeared and came back, and needs to be (re-)registered:
164 host:/sys/block/md5/bcache# echo /dev/sdh2 > /sys/fs/bcache/register
167 C) Corrupt bcache caching device crashes the kernel on startup/boot
169 You'll have to wipe the caching device, start the backing device without the
170 cache, and you can re-attach the cleaned up caching device then. This does
171 require booting with a kernel/rescue media where bcache is disabled
172 since it will otherwise try to access your device and probably crash
173 again before you have a chance to wipe it.
174 (or if you plan ahead, compile a backup kernel with bcache disabled and keep it
175 in your grub config for a rainy day)
176 If bcache is not available in the kernel, a filesystem on the backing device is
177 still available at an 8KiB offset. So either via a loopdev of the backing device
178 created with --offset 8K or by temporarily increasing the start sector of the
179 partition by 16 (512byte sectors).
181 This is how you wipe the caching device:
182 host:~# wipefs -a /dev/sdh2
183 16 bytes were erased at offset 0x1018 (bcache)
184 they were: c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
186 After you boot back with bcache enabled, you recreate the cache and attach it:
187 host:~# make-bcache -C /dev/sdh2
188 UUID: 7be7e175-8f4c-4f99-94b2-9c904d227045
189 Set UUID: 5bc072a8-ab17-446d-9744-e247949913c1
197 [ 650.511912] bcache: run_cache_set() invalidating existing data
198 [ 650.549228] bcache: register_cache() registered cache device sdh2
200 start backing device with missing cache:
201 host:/sys/block/md5/bcache# echo 1 > running
204 host:/sys/block/md5/bcache# echo 5bc072a8-ab17-446d-9744-e247949913c1 > attach
205 [ 865.276616] bcache: bch_cached_dev_attach() Caching md5 as bcache0 on set 5bc072a8-ab17-446d-9744-e247949913c1
208 D) Remove or replace a caching device
210 host:/sys/block/sda/sda7/bcache# echo 1 > detach
211 [ 695.872542] bcache: cached_dev_detach_finish() Caching disabled for sda7
213 host:~# wipefs -a /dev/nvme0n1p4
214 wipefs: error: /dev/nvme0n1p4: probing initialization failed: Device or resource busy
215 Ooops, it's disabled, but not unregistered, so it's still protected
217 We need to go and unregister it:
218 host:/sys/fs/bcache/b7ba27a1-2398-4649-8ae3-0959f57ba128# ls -l cache0
219 lrwxrwxrwx 1 root root 0 Feb 25 18:33 cache0 -> ../../../devices/pci0000:00/0000:00:1d.0/0000:70:00.0/nvme/nvme0/nvme0n1/nvme0n1p4/bcache/
220 host:/sys/fs/bcache/b7ba27a1-2398-4649-8ae3-0959f57ba128# echo 1 > stop
221 kernel: [ 917.041908] bcache: cache_set_free() Cache set b7ba27a1-2398-4649-8ae3-0959f57ba128 unregistered
224 host:~# wipefs -a /dev/nvme0n1p4
225 /dev/nvme0n1p4: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
228 E) dmcrypt and bcache
230 First setup bcache unencrypted and then install dmcrypt on top of /dev/bcache<N>
231 This will work faster than if you dmcrypt both the backing and caching
232 devices and then install bcache on top.
235 F) Stop/free a registered bcache to wipe and/or recreate it
236 (or maybe you need to free up all bcache references so that you can have fdisk
237 run and re-register a changed partition table, which won't work if there are any
238 active backing or caching devices left on it)
240 1) Is it present in /dev/bcache* ? (there are times where it won't be)
242 host:/sys/block/bcache0/bcache# echo 1 > stop
244 2) But if your backing device is gone, this won't work:
245 host:/sys/block/bcache0# cd bcache
246 bash: cd: bcache: No such file or directory
248 In this case, you may have to unregister the dmcrypt block device that
249 references this bcache to free it up:
250 host:~# dmsetup remove oldds1
251 bcache: bcache_device_free() bcache0 stopped
252 bcache: cache_set_free() Cache set 5bc072a8-ab17-446d-9744-e247949913c1 unregistered
254 This causes the backing bcache to be removed from /sys/fs/bcache and then it can
257 3) In other cases, you can also look in /sys/fs/bcache/:
258 host:/sys/fs/bcache# ls -l */{cache?,bdev?}
259 lrwxrwxrwx 1 root root 0 Mar 5 09:39 0226553a-37cf-41d5-b3ce-8b1e944543a8/bdev1 -> ../../../devices/virtual/block/dm-1/bcache/
260 lrwxrwxrwx 1 root root 0 Mar 5 09:39 0226553a-37cf-41d5-b3ce-8b1e944543a8/cache0 -> ../../../devices/virtual/block/dm-4/bcache/
261 lrwxrwxrwx 1 root root 0 Mar 5 09:39 5bc072a8-ab17-446d-9744-e247949913c1/cache0 -> ../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/ata10/host9/target9:0:0/9:0:0:0/block/sdl/sdl2/bcache/
263 The device names will show which UUID is relevant, cd in that directory
265 host:/sys/fs/bcache/5bc072a8-ab17-446d-9744-e247949913c1# echo 1 > stop
266 this will free up bcache references and let you reuse the partition for other
271 TROUBLESHOOTING PERFORMANCE
272 ---------------------------
274 Bcache has a bunch of config options and tunables. The defaults are intended to
275 be reasonable for typical desktop and server workloads, but they're not what you
276 want for getting the best possible numbers when benchmarking.
278 - Bad write performance
280 If write performance is not what you expected, you probably wanted to be
281 running in writeback mode, which isn't the default (not due to a lack of
282 maturity, but simply because in writeback mode you'll lose data if something
285 # echo writeback > /sys/block/bcache0/bcache/cache_mode
287 - Bad performance, or traffic not going to the SSD that you'd expect
289 By default, bcache doesn't cache everything. It tries to skip sequential IO -
290 because you really want to be caching the random IO, and if you copy a 10
291 gigabyte file you probably don't want that pushing 10 gigabytes of randomly
292 accessed data out of your cache.
294 But if you want to benchmark reads from cache, and you start out with fio
295 writing an 8 gigabyte test file - so you want to disable that.
297 # echo 0 > /sys/block/bcache0/bcache/sequential_cutoff
299 To set it back to the default (4 mb), do
301 # echo 4M > /sys/block/bcache0/bcache/sequential_cutoff
303 - Traffic's still going to the spindle/still getting cache misses
305 In the real world, SSDs don't always keep up with disks - particularly with
306 slower SSDs, many disks being cached by one SSD, or mostly sequential IO. So
307 you want to avoid being bottlenecked by the SSD and having it slow everything
310 To avoid that bcache tracks latency to the cache device, and gradually
311 throttles traffic if the latency exceeds a threshold (it does this by
312 cranking down the sequential bypass).
314 You can disable this if you need to by setting the thresholds to 0:
316 # echo 0 > /sys/fs/bcache/<cache set>/congested_read_threshold_us
317 # echo 0 > /sys/fs/bcache/<cache set>/congested_write_threshold_us
319 The default is 2000 us (2 milliseconds) for reads, and 20000 for writes.
321 - Still getting cache misses, of the same data
323 One last issue that sometimes trips people up is actually an old bug, due to
324 the way cache coherency is handled for cache misses. If a btree node is full,
325 a cache miss won't be able to insert a key for the new data and the data
326 won't be written to the cache.
328 In practice this isn't an issue because as soon as a write comes along it'll
329 cause the btree node to be split, and you need almost no write traffic for
330 this to not show up enough to be noticeable (especially since bcache's btree
331 nodes are huge and index large regions of the device). But when you're
332 benchmarking, if you're trying to warm the cache by reading a bunch of data
333 and there's no other traffic - that can be a problem.
335 Solution: warm the cache by doing writes, or use the testing branch (there's
336 a fix for the issue there).
339 SYSFS - BACKING DEVICE
340 ----------------------
342 Available at /sys/block/<bdev>/bcache, /sys/block/bcache*/bcache and
343 (if attached) /sys/fs/bcache/<cset-uuid>/bdev*
346 Echo the UUID of a cache set to this file to enable caching.
349 Can be one of either writethrough, writeback, writearound or none.
352 Writing to this file resets the running total stats (not the day/hour/5 minute
356 Write to this file to detach from a cache set. If there is dirty data in the
357 cache, it will be flushed first.
360 Amount of dirty data for this backing device in the cache. Continuously
361 updated unlike the cache set's version, but may be slightly off.
364 Name of underlying device.
367 Size of readahead that should be performed. Defaults to 0. If set to e.g.
368 1M, it will round cache miss reads up to that size, but without overlapping
369 existing cache entries.
372 1 if bcache is running (i.e. whether the /dev/bcache device exists, whether
373 it's in passthrough mode or caching).
376 A sequential IO will bypass the cache once it passes this threshold; the
377 most recent 128 IOs are tracked so sequential IO can be detected even when
378 it isn't all done at once.
381 If non zero, bcache keeps a list of the last 128 requests submitted to compare
382 against all new requests to determine which new requests are sequential
383 continuations of previous requests for the purpose of determining sequential
384 cutoff. This is necessary if the sequential cutoff value is greater than the
385 maximum acceptable sequential size for any single request.
388 The backing device can be in one of four different states:
390 no cache: Has never been attached to a cache set.
392 clean: Part of a cache set, and there is no cached dirty data.
394 dirty: Part of a cache set, and there is cached dirty data.
396 inconsistent: The backing device was forcibly run by the user when there was
397 dirty data cached but the cache set was unavailable; whatever data was on the
398 backing device has likely been corrupted.
401 Write to this file to shut down the bcache device and close the backing
405 When dirty data is written to the cache and it previously did not contain
406 any, waits some number of seconds before initiating writeback. Defaults to
410 If nonzero, bcache tries to keep around this percentage of the cache dirty by
411 throttling background writeback and using a PD controller to smoothly adjust
415 Rate in sectors per second - if writeback_percent is nonzero, background
416 writeback is throttled to this rate. Continuously adjusted by bcache but may
417 also be set by the user.
420 If off, writeback of dirty data will not take place at all. Dirty data will
421 still be added to the cache until it is mostly full; only meant for
422 benchmarking. Defaults to on.
424 SYSFS - BACKING DEVICE STATS:
426 There are directories with these numbers for a running total, as well as
427 versions that decay over the past day, hour and 5 minutes; they're also
428 aggregated in the cache set directory as well.
431 Amount of IO (both reads and writes) that has bypassed the cache
436 Hits and misses are counted per individual IO as bcache sees them; a
437 partial hit is counted as a miss.
441 Hits and misses for IO that is intended to skip the cache are still counted,
444 cache_miss_collisions
445 Counts instances where data was going to be inserted into the cache from a
446 cache miss, but raced with a write and data was already present (usually 0
447 since the synchronization for cache misses was rewritten)
450 Count of times readahead occurred.
454 Available at /sys/fs/bcache/<cset-uuid>
457 Average data per key in the btree.
460 Symlink to each of the attached backing devices.
463 Block size of the cache devices.
466 Amount of memory currently used by the btree cache
472 Symlink to each of the cache devices comprising this cache set.
474 cache_available_percent
475 Percentage of cache device which doesn't contain dirty data, and could
476 potentially be used for writeback. This doesn't mean this space isn't used
477 for clean cached data; the unused statistic (in priority_stats) is typically
481 Clears the statistics associated with this cache
484 Amount of dirty data is in the cache (updated when garbage collection runs).
487 Echoing a size to this file (in human readable units, k/M/G) creates a thinly
488 provisioned volume backed by the cache set.
492 These determines how many errors we accept before disabling the cache.
493 Each error is decayed by the half life (in # ios). If the decaying count
494 reaches io_error_limit dirty data is written out and the cache is disabled.
497 Journal writes will delay for up to this many milliseconds, unless a cache
498 flush happens sooner. Defaults to 100.
501 Percentage of the root btree node in use. If this gets too high the node
502 will split, increasing the tree depth.
505 Write to this file to shut down the cache set - waits until all attached
506 backing devices have been shut down.
509 Depth of the btree (A single node btree has depth 0).
512 Detaches all backing devices and closes the cache devices; if dirty data is
513 present it will disable writeback caching and wait for it to be flushed.
515 SYSFS - CACHE SET INTERNAL:
517 This directory also exposes timings for a number of internal operations, with
518 separate files for average duration, average frequency, last occurrence and max
519 duration: garbage collection, btree read, btree node sorts and btree splits.
521 active_journal_entries
522 Number of journal entries that are newer than the index.
525 Total nodes in the btree.
528 Average fraction of btree in use.
531 Statistics about the auxiliary search trees
533 btree_cache_max_chain
534 Longest chain in the btree node cache's hash table
537 Counts instances where while data was being read from the cache, the bucket
538 was reused and invalidated - i.e. where the pointer was stale after the read
539 completed. When this occurs the data is reread from the backing device.
542 Writing to this file forces garbage collection to run.
544 SYSFS - CACHE DEVICE:
546 Available at /sys/block/<cdev>/bcache
549 Minimum granularity of writes - should match hardware sector size.
552 Sum of all btree writes, in (kilo/mega/giga) bytes
557 cache_replacement_policy
558 One of either lru, fifo or random.
561 Boolean; if on a discard/TRIM will be issued to each bucket before it is
562 reused. Defaults to off, since SATA TRIM is an unqueued command (and thus
566 Size of the freelist as a percentage of nbuckets. Can be written to to
567 increase the number of buckets kept on the freelist, which lets you
568 artificially reduce the size of the cache at runtime. Mostly for testing
569 purposes (i.e. testing how different size caches affect your hit rate), but
570 since buckets are discarded when they move on to the freelist will also make
571 the SSD's garbage collection easier by effectively giving it more reserved
575 Number of errors that have occurred, decayed by io_error_halflife.
578 Sum of all non data writes (btree writes and all other metadata).
581 Total buckets in this cache
584 Statistics about how recently data in the cache has been accessed.
585 This can reveal your working set size. Unused is the percentage of
586 the cache that doesn't contain any data. Metadata is bcache's
587 metadata overhead. Average is the average priority of cache buckets.
588 Next is a list of quantiles with the priority threshold of each.
591 Sum of all data that has been written to the cache; comparison with
592 btree_written gives the amount of write inflation in bcache.