In this guide: 14 sections
- What to do, in order
- Four builds, end to end
- Three things are called caching, and only one is safe to lose
- Will a cache help? The access pattern decides
- ZFS L2ARC: a read cache paid for in RAM
- ZFS special vdev: metadata on flash, for good
- ZFS SLOG: only for synchronous writes
- Linux without ZFS: LVM cache, bcache and dm-writecache
- Windows: Storage Spaces tiering, and what is left
- NAS appliances: Synology, QNAP, unRAID, TrueNAS
- Choosing the NVMe: write budget first, then power-loss protection
- Checking that it worked
- Power cuts, dead drives and removal
- When caching is the wrong answer
For most home servers and NAS boxes, an NVMe cache in front of the hard drives is the third purchase, not the first: more RAM comes first, then filesystem metadata on a mirrored pair of small SSDs. Before choosing any software, decide which of three things you need, a read cache, a write cache or a tier: a read cache that dies costs nothing but speed, while a write cache or a tier holds the only copy of your data and needs a mirror and power-loss protection. On ZFS that order is the ARC, then a mirrored special vdev, then an L2ARC only when the data you re-read outgrows any RAM the board can hold; a SLOG helps synchronous writes and nothing else. On plain Linux the safe start is LVM cache in writethrough mode, which can lose its SSD and lose no data. For a media library or a backup target, a cache does nothing at all.
What to do, in order
Stop at the first step that fixes the problem.
- Name the slow operation. Directory walks,
find, rsync scans and scrubs are metadata. VM disks and databases are small random reads. NFS exports and database commits are synchronous writes. A film copy or a backup is sequential, and no cache speeds it up. - Check the RAM cache first. Run
cachestat 1on Linux or read the ARC hit ratio on ZFS. If the board has empty slots, memory that fits it is the cheaper fix. - On ZFS, put metadata on a mirrored special vdev before buying any L2ARC. Losing it loses the pool, and on a raidz pool it can never be removed: read the special vdev section before running
zpool add. - Buy a SLOG only if
zilstatshows synchronous writes. Most home pools issue none; when one does, 8 to 32 GB of a drive with power-loss protection is enough. - Add an L2ARC only when the re-read working set outgrows any ARC the chassis can hold, records are large, and the device can take at least about 2.6 TiB of writes a day (the nominal pace; busy feeds can run several times faster).
- Off ZFS, start with LVM cache in writethrough. It attaches to existing data, and a dead SSD loses nothing.
- Treat any write cache as storage. Mirror it, buy power-loss protection, rehearse the drain.
- Measure for a week, then grow, reading the hit ratio beside the I/O rate, the p99 latency and the NVMe’s
data_units_written.
Which setup fits your box
| Your box and its pain | What helps | What does not |
|---|---|---|
ZFS NAS with slow find, scrubs and snapshot listings |
A mirrored special vdev, metadata only | An L2ARC, which never sees metadata until it has been read once |
| ZFS NAS serving NFS, VMs or a database | A SLOG with power-loss protection, once zilstat shows sync traffic |
A consumer NVMe SLOG |
| ZFS box with every DIMM slot full and a re-read set bigger than RAM | An L2ARC on datasets with large records | An L2ARC on a box that should have bought DIMMs |
| Linux file server with slow photo libraries and directory walks | LVM cache in writethrough | Writeback on one unmirrored SSD |
| Linux workstation doing small random writes to rebuildable data | bcache writeback on two mirrored PLP SSDs | A writeback cache on one consumer drive |
| Linux host bound by fsync latency, no ZFS | dm-writecache, mirrored, with PLP | Expecting it to speed up reads |
| Windows desktop | More RAM; PrimoCache if you accept its write risk | ReadyBoost, Intel SRT, Optane Memory |
| Synology | A read-only cache on one SSD, or a read-write cache on two | Any cache on a camera NAS or a single-stream video box |
| unRAID | A pool for appdata, Docker and VM disks | Expecting the pool to cache reads |
| Media library or backup target | Capacity at the lowest price per terabyte | Any cache at all |
Starting sizes, to measure and then move
No vendor publishes a table like this; it is a judgment to start from, not a rule.
| Build | HDD raw | Cache | Notes |
|---|---|---|---|
| Desktop, 4-bay NAS | 4 to 16 TB | 250 to 500 GB | writethrough, unmirrored |
| 8 to 24 bay server | 50 to 200 TB | 1 to 2 TB | mirror anything writeback |
| Hypervisor node | 200 TB and up | 2 to 4 TB | PLP mandatory, 1 DWPD or better |
On ZFS, split the middle row: a mirrored special vdev sized from zdb -bbb, plus an unmirrored L2ARC that costs nothing to lose.
Four builds, end to end
Read the removal procedure before you run the creation commands. Recipes 1 and 3 can be undone, recipe 3 only after its dirty data is flushed; recipe 4 destroys the volume on the way out; recipe 2 cannot be undone at all on a raidz pool.
Recipe 1: Linux home server, LVM cache in writethrough
For: a file server whose pain is read latency - photo libraries, media metadata, a Samba share that stalls on directory walks. Not for: sync-heavy work, since writethrough completes nothing until the hard drive has it.
Hardware: two CMR hard drives of equal size, mirrored, and one NVMe SSD in M.2. Writethrough never holds the only copy of anything, so a single consumer drive from the M.2 NVMe listing is defensible here as nowhere else in this guide. Partition 80 to 90 percent of it and leave the rest as spare area.
pvcreate /dev/sda /dev/sdb /dev/nvme0n1p1
vgcreate vg0 /dev/sda /dev/sdb /dev/nvme0n1p1
# mirrored origin across the two spindles (4 TB disks)
lvcreate --type raid1 -m 1 -n main -L 3.5T vg0 /dev/sda /dev/sdb
# cache pool: separate data and metadata sub-LVs
lvcreate --type cache-pool -L 180G -n fast --chunksize 256K vg0 /dev/nvme0n1p1
lvconvert --type cache --cachepool vg0/fast \
--cachemode writethrough --cachepolicy smq vg0/main
mkfs.xfs /dev/vg0/main
Verify:
lvs -o lv_name,cache_mode,cache_policy,chunksize,\
cache_used_blocks,cache_read_hits,cache_read_misses vg0/main
dmsetup status --target cache --noflush /dev/mapper/vg0-main
Check chunksize: LVM raises it to keep a cache under 1,000,000 chunks. Always pass --noflush, or every status read commits metadata and a one-second monitoring poll writes to the NVMe every second, forever.
Removal:
lvconvert --splitcache vg0/main # flush, detach, keep the fast LV
lvconvert --uncache vg0/main # flush, detach, delete the fast LV
If the NVMe dies first, no data is lost, but the LV returns errors until the cache is detached, and lvconvert --uncache refuses while the PV is missing. The community route is vgreduce --removemissing --force first; it varies by lvm2 version, so rehearse it on a loopback pair.
Recipe 2: ZFS pool with a mirrored special vdev
For: a pool whose slow operations are metadata - find, rsync traversals, snapshot listing, scrub and resilver, backup scans. This is the highest-value build in this guide. Not for: anyone who cannot mirror it. Losing it loses the pool.
Hardware: six hard drives in raidz2 (drives for a NAS covers choosing them) plus three NVMe SSDs with power-loss protection as a three-way special mirror, matching raidz2’s two-disk tolerance. Given only two, zpool add demands -f, which the redundancy section says to read as a no. Size the special vdev from measurement:
zdb -bbb tank | less # space by block type; read off the metadata total
Correct that total as sizing one describes, with generous headroom: the vdev cannot be shrunk.
zpool create -o ashift=12 tank raidz2 \
/dev/disk/by-id/ata-HDD1 /dev/disk/by-id/ata-HDD2 \
/dev/disk/by-id/ata-HDD3 /dev/disk/by-id/ata-HDD4 \
/dev/disk/by-id/ata-HDD5 /dev/disk/by-id/ata-HDD6
zpool add tank special mirror \
/dev/disk/by-id/nvme-SSD_A-part1 /dev/disk/by-id/nvme-SSD_B-part1 \
/dev/disk/by-id/nvme-SSD_C-part1
zfs create tank/media
zfs set recordsize=1M tank/media
zfs create tank/vms
zfs set special_small_blocks=32K tank/vms
special_small_blocks defaults to 0, meaning metadata only; raise it per dataset, never pool-wide, for the reason given in the special vdev section.
Verify:
zpool list -v tank # per-vdev capacity; watch the special vdev fill
zpool status tank # all mirror members ONLINE
Removal is zpool remove tank mirror-1, using the name zpool status prints, and it works only if no top-level vdev is raidz or draid, ashift is uniform, device_removal is enabled and encryption keys are loaded. On the raidz2 pool above it can never be removed, which is why the mirror is not optional.
Recipe 3: Linux workstation, bcache in writeback
For: a single-user machine doing heavy small random writes - compile trees, CI workspaces, container layers, a scratch dataset you can rebuild. Not for: anything whose loss would matter; writeback is off by default because “in writeback mode you’ll lose data if something happens to your SSD”.
Hardware: one hard drive from the hard drive listings and two NVMe drives with power-loss protection, mirrored with md, because bcache has no native cache mirroring. Power-loss protection is the filter that matters, for the reason in the SLOG section; the enterprise class marks duty rating, so confirm the protection on the exact model’s datasheet.
mdadm --create /dev/md10 --level=1 --raid-devices=2 \
/dev/nvme0n1p1 /dev/nvme1n1p1
make-bcache -B /dev/sdc -C /dev/md10 -b 512k
# newer packages: bcache make -B /dev/sdc -C /dev/md10 -b 512k
mkfs.ext4 /dev/bcache0
echo writeback > /sys/block/bcache0/bcache/cache_mode
echo always > /sys/block/sdc/bcache/stop_when_cache_set_failed
Set the bucket size with -b explicitly: the bcache-tools source defaults to 512 KiB while the 1.0.8 man page still says 128k. Then watch:
cat /sys/block/bcache0/bcache/dirty_data # bytes at risk
cat /sys/block/bcache0/bcache/stats_hour/cache_hit_ratio
cat /sys/fs/bcache/CSET-UUID/internal/cutoff_writeback # 40
cat /sys/fs/bcache/CSET-UUID/internal/cutoff_writeback_sync # 70
writeback_percent defaults to 10 and is silently clamped to 40. Past 40 percent of the cache in use, only sync and metadata writes are still written back; past 70 percent, writeback stops. That clamp is why “my writeback cache stopped absorbing writes”.
Removal, in this order:
echo 1 > /sys/block/sdc/bcache/detach # flushes dirty data first
cat /sys/block/bcache0/bcache/dirty_data # must read 0 before continuing
echo 1 > /sys/fs/bcache/CSET-UUID/stop
wipefs -a /dev/md10
Detach alone leaves the cache registered; stop frees it. If the cache is already dead, echo 1 > /sys/block/sdc/bcache/running forces the backing device up alone; mount it read-only and expect damage.
Recipe 4: Windows tiered Storage Space, in PowerShell
For: an NTFS volume on Windows Server whose hot region is a stable minority of a large dataset. Not for: live acceleration, since data moves between tiers at 1:00 a.m. The virtual disk must use fixed provisioning, and both tiers take the same column count, so a four-column two-way mirror wants eight SSDs and eight hard drives.
$sub = (Get-StorageSubSystem -FriendlyName "Windows Storage*").FriendlyName
New-StoragePool -FriendlyName Pool1 -StorageSubSystemFriendlyName $sub `
-PhysicalDisks (Get-PhysicalDisk -CanPool $true)
# set MediaType by hand on any disk that reports Unspecified
Get-PhysicalDisk | Where-Object MediaType -eq Unspecified |
Set-PhysicalDisk -MediaType HDD
New-StorageTier -StoragePoolFriendlyName Pool1 -FriendlyName SSDTier -MediaType SSD
New-StorageTier -StoragePoolFriendlyName Pool1 -FriendlyName HDDTier -MediaType HDD
New-Volume -StoragePoolFriendlyName Pool1 -FriendlyName Data -DriveLetter D `
-ResiliencySettingName Mirror -ProvisioningType Fixed `
-StorageTiers (Get-StorageTier -FriendlyName SSDTier),
(Get-StorageTier -FriendlyName HDDTier) `
-StorageTierSizes 200GB, 4TB -FileSystem NTFS
Verify, and run the first optimization by hand:
Get-StorageTier | Format-Table FriendlyName, MediaType, Size
Optimize-Volume -DriveLetter D -TierOptimize
defrag D: /g /h /# # /h runs at normal priority, /# writes the report
Get-FileStorageTier -VolumeDriveLetter D
schtasks /change /tn "\Microsoft\Windows\Storage Tiers Management\Storage Tiers Optimization" /ri 360
The report gives the share of I/O the SSD tier served and what tier size would reach a target: a measured miss-ratio curve, which Linux has nothing comparable to. Microsoft recommends running the task at most every 6 hours.
Removal destroys the volume, so back up first:
Remove-VirtualDisk -FriendlyName Data
Remove-StorageTier -FriendlyName SSDTier
Remove-StorageTier -FriendlyName HDDTier
Remove-StoragePool -FriendlyName Pool1
Windows 11 and 10 do not expose tiers in Control Panel, and Microsoft documents no supported client-edition tiering scenario.
Three things are called caching, and only one is safe to lose
A 7200 rpm drive’s average rotational latency is 4.16 ms, and it was 4.16 ms twenty years ago: half of one 8.33 ms revolution. With an average seek added, one random read costs about 12 ms, against roughly 80 to 120 µs for a 4K read on a consumer TLC NVMe. Capacity per spindle has grown more than twentyfold since; seeks per second have not. That gap is what a cache buys back. Sequential speed is not the problem: an IronWolf Pro 20 TB is rated at up to 285 MB/s, faster than most home networks.
The three mechanisms differ in the one respect that matters at three in the morning: where the only copy of your data lives.
| What the fast device holds | Losing it costs | |
|---|---|---|
| Read cache | A second copy of hot blocks | Performance only |
| Write cache | The only copy, until destaged | The un-destaged writes |
| Tiering | The only copy, permanently | That data, until restored |
A read cache risks nothing but the money, the wear and the RAM it uses. A read error on a ZFS L2ARC is reissued to the pool, and zpool remove tank nvme0n1 takes the device out with no ceremony. dm-cache in writethrough and bcache in writethrough or writearound make the same promise.
A write cache holds acknowledged writes that exist nowhere else until they are destaged to the hard drive; lvmcache(7) says plainly that “the loss of a cache device can result in lost data.” A writeback cache in front of a redundant array turns it into a non-redundant one, because the NVMe is a single point of failure upstream of all the redundancy you paid for.
A tier is not a cache at all. Blocks routed to a ZFS special vdev exist only there, so OpenZFS says its redundancy “should match the redundancy of the other normal devices in the pool”. Storage Spaces tiering and bcachefs background_target likewise move data rather than copy it. With a tier, the device choice is a durability decision before it is a speed one.
Will a cache help? The access pattern decides
A cache is a bet that your reuse distance is shorter than your cache is big. Lose the bet and you have bought a wear-out mechanism in the write path. What decides it is the count of distinct blocks touched in a window, not the volume of I/O: a workload reading 2 TB a day across an 80 GB footprint caches almost perfectly, while one reading 200 GB a day scattered over 6 TB does not. Drift matters too: in Microsoft’s words, if the working set “drifts too quickly, read cache misses increase”. And the fall-off is not gentle. A footprint read in a repeating cycle evicts every block just before it is needed, so a cache that was fine at 400 GB of footprint can collapse at 550 GB.
Hit ratio is the number that matters
L_eff = h * L_fast + (1 - h) * L_slow
S = L_slow / L_eff
L_slow = 12 ms 7200 rpm, QD1 random 4K, DERIVED: Seagate gives 4.16 ms
average rotational latency (Exos X24 manual) and 168 random
read IOPS at 4K QD16 (datasheet). QD16 lets the drive
reorder; assuming QD1 reaches about half, 84 IOPS, one read
costs about 12 ms. Substitute your own measurement.
L_fast = 0.1 ms consumer TLC NVMe, 4K read, QD1. Vendors publish IOPS,
not QD1 latency, so 0.1 ms is an estimate.
h = 0.50 L_eff = 0.050 + 6.000 = 6.050 ms S = 2.0x
h = 0.80 L_eff = 0.080 + 2.400 = 2.480 ms S = 4.8x
h = 0.90 L_eff = 0.090 + 1.200 = 1.290 ms S = 9.3x
h = 0.95 L_eff = 0.095 + 0.600 = 0.695 ms S = 17.3x
h = 0.99 L_eff = 0.099 + 0.120 = 0.219 ms S = 54.8x
h = 0.999 L_eff = 0.100 + 0.012 = 0.112 ms S = 107.2x
The curve is hyperbolic. Moving from 50 to 90 percent buys 4.7 times; 90 to 99 percent buys another 5.9 times. Think in misses, not hits: a 95 percent hit ratio is a 5 percent miss rate, and 5 percent of 12 ms is 600 µs, six times slower than the flash. Report p99, not the mean, because at h = 0.99 everything above the 99th percentile is a hard disk access. And 99 percent of 20 IOPS is not a result: quote the I/O rate beside the ratio.
What pays for the flash
| Benefits | Mechanism |
|---|---|
| Filesystem metadata | Tiny, hot, random, on the critical path of everything. find, ls -lR, rsync traversals and backup scans collapse. The highest-leverage target there is. |
| VM images and VDI | A small hot region inside an enormous file. Microsoft’s Windows Server 2012 R2 Storage Tiers documentation: “if only 30 percent of the data on a virtual hard disk is ‘hot’, only that 30 percent moves to your solid-state drives”. |
| OLTP databases | Random 4K to 16K reads over an index working set far smaller than the table. Caveat: the buffer pool already holds the hottest data, so the block cache sees a larger, colder second tier. |
| Compile trees, package caches, CI workspaces | Vast numbers of small files, near-total reuse, a footprint of a few GB. |
| Mail spools, container layers, Git object stores | The same shape. |
| Random write bursts | A write cache derandomizes before destaging, converting seeks into streams. |
Sequential streams, backup targets that are written once and never read, full scans and footprints larger than the cache gain nothing; when caching is the wrong answer covers why.
Measure your own workload first
Start above the block layer, because a miss the page cache absorbed is not your problem:
cachestat 1 # page cache hits/misses, from bcc
bitesize # IO size histogram: 4K random or 1 MiB streams?
biolatency -D 1 # per-device latency histogram
iostat -x 1 # r_await, w_await, rareq-sz, wareq-sz, aqu-sz
Ignore %util on an NVMe device, which iostat’s own man page says “does not reflect their performance limits”.
Most stacks refuse to cache streams on purpose, because one 10 GB copy would otherwise evict the hot set. bcache bypasses runs past sequential_cutoff, 4 MiB by default; ZFS’s l2arc_noprefetch=1 keeps unused prefetched blocks off the cache device; tiered Storage Spaces skips its 1 GB write-back cache for writes over 256 KB (as documented for Windows Server 2012 R2). dm-cache has no documented sequential bypass since mq was removed in Linux 4.6, so keep big sequential jobs on a separate volume.
ZFS L2ARC: a read cache paid for in RAM
On ZFS the cache is the RAM, and the NVMe device is the consolation prize. The ARC lives in memory, and a device added with zpool add tank cache /dev/nvme0n1 is a second-level spill for blocks the ARC was about to throw away. OpenZFS: “Before adding any cache device, check whether the ARC is simply too small.”
Through 2.2, Linux capped the ARC at half of RAM; from OpenZFS 2.3 the cap is the larger of RAM minus 1 GiB and five eighths of RAM, so a 64 GiB box gives it 63 GiB, not 32. On a second-hand server board the first purchase is usually registered ECC memory that fits the board, and how much to fit is the subject of RAM caching and, platform by platform, of how much RAM for a NAS or homelab. The “1 GB of RAM per TB” rule appears nowhere in OpenZFS documentation; the current TrueNAS hardware guide gives a base of 8 GB, plus roughly 5 GB per TB with deduplication on, plus about 1 GB for every 50 GB of L2ARC.
What L2ARC holds, and what it never sees
L2ARC is a read cache only: “The L2ARC does not store dirty content.” No write is ever at risk on it, so it needs no power-loss protection and no mirror. The feed thread copies blocks the ARC is about to evict, so blocks never read cannot be there: a scrub, a resilver, a backup walk or a find over cold metadata gets nothing from it. Set secondarycache=metadata on a media dataset to keep streams out.
The header tax, and where it stops
Every block held in L2ARC but not in ARC keeps a header in ARC, about 96 bytes on x86-64 for the OpenZFS 2.2 to 2.4 layout; the TrueNAS guide uses the same figure. The running total is the l2_hdr_size kstat, so measure it rather than estimate it.
Header cost per TiB of L2ARC content, at 96 B per record.
Record size here is the PHYSICAL (post-compression) block size,
because that is what lands on the device.
128 KiB blocks: 1 TiB / 131072 B = 8,388,608 recs x 96 = 768 MiB
32 KiB blocks: 1 TiB / 32768 B = 33,554,432 recs x 96 = 3 GiB
16 KiB blocks: 1 TiB / 16384 B = 67,108,864 recs x 96 = 6 GiB
8 KiB blocks: 1 TiB / 8192 B = 134,217,728 recs x 96 = 12 GiB
4 KiB blocks: 1 TiB / 4096 B = 268,435,456 recs x 96 = 24 GiB
TrueNAS's "1 GB per 50 GB of L2ARC" is 2%, which back-solves to an
assumed average physical block of about 4,800 B (4.7 KiB). It is a worst-case
small-block figure, not a universal one.
Those headers are not evictable under memory pressure, so l2arc_meta_percent, default 33, stops the feed once they pass a third of the ARC target. A 16 GiB ARC tops out at about 59 million headers: 901 GiB of content at 16 KiB physical blocks, 7.04 TiB at 128 KiB. A 2 TB cache device in front of that ARC on a database pool at 16 KiB records never fills past about 900 GiB. Your RAM sizes your L2ARC, and your block size decides how far it stretches.
How fast it fills, and what that costs the SSD
l2arc_write_max caps what one feed writes, and l2arc_feed_secs, default 1, sets the seconds between feeds. The defaults changed once, with OpenZFS #15457 in 2.2.7 (December 2024):
| OpenZFS release | l2arc_write_max |
l2arc_write_boost |
l2arc_headroom |
l2arc_noprefetch |
|---|---|---|---|---|
| 2.2.6 and earlier, 2.1 series to 2.1.16 | 8 MiB | 8 MiB | 2 | 1 |
| 2.2.7 to 2.4.4, the latest (August 21, 2026) | 32 MiB | 32 MiB | 8 | 1 |
Source: zfs.4 and module/zfs/arc.c at tags zfs-2.2.6, zfs-2.2.7 and zfs-2.4.4 of the OpenZFS repository, checked October 15, 2026. l2arc_headroom sets how far through the ARC lists a feed searches, as “a multiplier of l2arc_write_max”; 0 allows “the full length of ARC lists to be searched”. The unreleased development branch raises the write size to 64 MiB, drops l2arc_write_boost and adds l2arc_dwpd_limit; no release, 2.4.4 included, has any of that.
So the nominal pace is 32 MiB/s, and it is neither a ceiling nor a steady state. Whenever a feed wrote more than half its quota, warm or cold, l2arc_feed_again cuts the gap to 200 ms; while the ARC is cold, l2arc_write_boost is added on top. And a feed writes only what the ARC is evicting, so real wear is usually below the nominal 2.64 TiB a day.
1 TiB of L2ARC content at the 2.2.7+ write size of 32 MiB per feed,
feeds 1 s apart (the quiet-list interval, not a hard ceiling):
1 TiB / 32 MiB/s = 32,768 s = 9.1 hours of CONTINUOUS feeding
cold, with boost: 64 MiB/s = 4.6 hours
on 2.2.6 and earlier (8 MiB): = 36.4 hours
l2arc_feed_again (default 1) drops the gap to l2arc_feed_min_ms
= 200 ms while the lists are busy, so up to 5 feeds a second:
1 TiB / 160 MiB/s = 6,554 s = 1.8 hours
The 1 s interval read as a wear budget (the 200 ms interval can
multiply it by up to five):
32 MiB/s x 86,400 s = 2.64 TiB written per day
on a 1 TiB cache device that is about 2.6 DWPD, indefinitely
A consumer TLC drive rated at 0.33 drive writes per day does not survive that pace, which argues for a write-rated NVMe device over the fastest one on the shelf, or for lowering l2arc_write_max.
Persistent L2ARC has been on by default since OpenZFS 2.0, so the cache comes back warm after a reboot. A device re-added with zpool add is overwritten and starts empty; one brought back with zpool online is restored.
The verdict: an L2ARC earns its slot only on the three conditions in the checklist. A comment in OpenZFS’s unreleased development code puts the floor at twice the ARC’s maximum size, below which an L2ARC either overwrites itself in a cycle or duplicates the ARC. Short of all that, spend the money on memory.
ZFS special vdev: metadata on flash, for good
This is the highest-value flash in a pool of spinning disks, and one of the few devices that takes the pool with it when it dies. OpenZFS: “It is not a cache. The blocks routed there exist only there.”
From zpoolconcepts(7): “By default, this includes all metadata, the indirect blocks of user data, intent log (in absence of separate log device), and deduplication tables.” A dedicated SLOG always takes the intent log ahead of it. When the special class fills, new allocations spill back to the normal vdevs, so a full special vdev slows down quietly rather than failing.
special_small_blocks, and the trap inside it
The property is per dataset and defaults to 0, meaning off: metadata goes to the special vdev either way, and this only adds small user data. Through 2.3 it takes zero or a power of two from 512 up to 1 MiB; 2.4 allows up to 16 MiB. The trap is that it compares the size “after compression”, so a 128 KiB record that compresses to 20 KiB counts as 20 KiB, and a threshold near recordsize routes almost a whole well-compressing dataset onto the special vdev. Small blocks stop at 75 percent full (zfs_special_class_metadata_reserve_pct, default 25), but metadata alone can still fill it. Leave it at 0, measure, then raise it per dataset: 16K on a source tree or a maildir, 0 on a media dataset.
Why metadata on flash beats an L2ARC
A special vdev holds every new metadata block from the moment it is written: no warm-up, no ARC header cost, and blocks served on first access to a scrub, a resilver, find, zfs list or a du across ten million inodes, all of which, on metadata not yet read, miss an L2ARC by construction. OpenZFS calls it “usually a larger and more reliable win than adding an L2ARC”. Time a find and a scrub before and after to see the size of the win on your own pool.
Sizing one
Upstream calls sizing “workload-dependent and hard to predict from first principles”, and the “0.3 percent of pool” figure that circulates has no first-party source. Work the floor: each record has a 128-byte block pointer and most metadata is kept twice by default, so 10 TiB of data at recordsize=128K is 83,886,080 records and 20 GiB of block pointers with both copies, or 2.5 GiB at recordsize=1M. With dnodes, directories and space maps, a 10 TiB pool is still in tens of GiB, not hundreds. Then read your own pool:
Measure: zdb -bbb tank -> block-type breakdown, metadata total
Two corrections to that output, both community-sourced, not documented:
PSIZE under-counts - it ignores redundant metadata copies (~2x)
ASIZE over-counts - it includes raidz parity overhead
Worked shape, not a prediction. Suppose zdb reports:
metadata, PSIZE 180 GiB
x2 for the redundant metadata copies 360 GiB
/ 0.75 for the 25% metadata reserve floor 480 GiB
+ small blocks, if you enable them + your own figure
+ headroom, because you cannot shrink it
-> a mirrored pair of 1 TB NVMe, not a pair of 480 GB
Redundancy and removal: the part to read twice
“A special vdev must be at least as redundant as the pool’s normal vdevs. Losing it loses the pool, exactly as losing any other top-level vdev would. Never add one as a single device to a redundant pool.” Read literally, that means a three-way mirror under raidz2 and a four-way under raidz3, though upstream states the principle rather than that mapping. And “on a raidz pool, a special vdev can never be removed.” If zpool add demands -f because the special vdev is less redundant than the data vdevs, that is the pool telling you not to.
Existing blocks are never migrated: the special vdev takes newly written data only, and neither a scrub nor a resilver moves anything. Retrofitting means rewriting the data with zfs send or waiting for it to turn over, which on an archive pool is never.
ZFS SLOG: only for synchronous writes
A separate log device is not a write cache, and on a pool with no synchronous writes it changes nothing measurable. Every pool already has a ZFS Intent Log, by default “allocated from blocks within the main pool”; a log vdev only moves it onto something with lower commit latency. In normal operation the ZIL is written and never read; it is replayed only after a crash.
Asynchronous writes never touch it. They are gathered in memory and written with the next transaction group, every 5 seconds by default (zfs_txg_timeout) or sooner if the dirty-data limit fills. The workloads that issue synchronous writes are NFS exports, databases committing transactions and VM datastores set to sync. Most home and media pools, backup targets and desktops are asynchronous throughout, and for them the money belongs in RAM or a special vdev.
Find out before you buy
zilstat ships with OpenZFS from 2.2 onward. An hour under the real workload settles it:
# Run the real workload for an hour, then read the two fields that decide it:
grep -E 'zil_itx_metaslab_(normal|slog)_bytes' /proc/spl/kstat/zfs/zil
zil_itx_metaslab_normal_bytes ZIL traffic landing in the main pool
zil_itx_metaslab_slog_bytes ZIL traffic landing on a log vdev
# Both near zero under production load -> a SLOG will do nothing for you.
# normal_bytes large, no log vdev present -> a SLOG is the right purchase.
Sizing, which is far smaller than people expect
The device holds only the synchronous writes that could still be lost, a couple of transaction groups’ worth:
Sustained synchronous write rate (your figure) 500 MB/s
zfs_txg_timeout 5 s
Bytes accumulated before one commit 500 x 5 = 2.5 GB
Two transaction groups outstanding 2.5 x 2 = 5 GB
A saturated 10 GbE link of nothing but sync writes, about 1.1 GB/s, lands near 11 GB. The TrueNAS hardware guide agrees: “a high-endurance, low-latency device between 8 GB and 32 GB is adequate for most modern networks”. Partition 8 to 32 GB of a larger drive, leave the rest unwritten after a blkdiscard, and the controller uses it as spare area.
Power-loss protection is the product
The SLOG exists to make one promise: this write is durable. A drive that acknowledges a FLUSH before the data is safe breaks that promise silently, and shows it only in a power cut. Published third-party fsync measurements (fio with O_DIRECT, fsync after a 16 KB write) put a Samsung 990 Pro near 2,970 µs and a Crucial T500 near 890 µs, against roughly 12 µs for a Solidigm D7-P5520 and under 2 µs for a Samsung PM9A3; read the ranking, not the decimals. A consumer NVMe SLOG at 3 ms is competing with the 4.16 ms disks it was bought to rescue, and losing on correctness as well. Filter for the protection, not the brand: start from enterprise SSDs, a class that marks duty rating, then confirm the protection on the exact model’s datasheet.
On current OpenZFS a dead log device is an inconvenience: the pool falls back to its in-pool ZIL and zpool remove takes the device out. Data is lost only if the device dies and the system crashes in the same moment, which is what mirroring a SLOG covers. The forum advice that a dead SLOG takes the pool with it dates from pools older than ZFS pool version 19, when log device removal arrived.
Linux without ZFS: LVM cache, bcache and dm-writecache
LVM cache (dm-cache)
The raw dm-cache target defaults to writeback; LVM defaults to writethrough. Same target, opposite defaults, and the difference is whether a dead NVMe costs performance or data.
Most dm-cache tuning advice on the web is a no-op. Tutorials still set sequential_threshold 512 and random_threshold 4, which belonged to the mq policy deleted in Linux 4.6; mq is now an alias for smq, and lvmcache(7) says such settings are “silently ignored”.
Check the chunk size. LVM’s default is 64 KiB, but allocation/cache_pool_max_chunks defaults to 1,000,000 and LVM raises the chunk size to stay under it, so a terabyte-class cache lands in the megabyte range. Every promotion writes a whole chunk:
4 KiB write, promotion writes one whole chunk
64 KiB chunk -> 16x amplification
1 MiB chunk -> 256x
2 MiB chunk -> 512x
Run lvs -o+chunksize after conversion. One hard incompatibility from lvmcache(7): a cache on a device with 4096-byte logical blocks cannot serve a filesystem made with 512, and “the main LV will likely fail to mount”. Check xfs_info for sectsz= first.
Before choosing writeback, know three behaviors. After a crash, “all cache blocks will be assumed dirty when restarted”, so the next detach drains the whole cache. On a server that is never idle, a writeback cache never drains by itself; the reliable drains are lvchange --cachepolicy cleaner, switching to writethrough, or lvconvert --splitcache. And migration_threshold is not a bandwidth cap but a limit on migrations in flight: the kernel default of 2048 sectors (1 MiB) allows one at 1 MiB chunks, and LVM sets a floor of eight chunks. If the cache warms at a crawl, raise it with lvchange --cachesettings 'migration_threshold=8192' vg/main (and 'migration_threshold=default' to undo it).
bcache
bcache is not dm-cache with a different spelling. It is a small log-structured allocator that writes in erase-block-sized buckets, “designed to avoid random writes at all costs”, so a 4 KiB write never triggers a chunk-sized promotion. Its sharp edges: formatting is required, because the backing device gets a superblock and an 8 KiB data offset; there is no native mirroring, since multiple caches per set “isn’t supported yet”; and dirty data on a lost cache device is lost, with “massive filesystem corruption” if you force the backing device up after a writeback failure. Format every slow device as a backing device from the start, as the documentation advises, and flash can be added later.
| Mode | Cache device lost |
|---|---|
writethrough (default) |
no data loss; discard the device |
writearound |
no data loss; reads-only caching |
writeback |
dirty data gone, backing device stale |
Its instruments beat dm-cache’s: stats_hour and its siblings hold decaying hit ratios, and bypassed shows the sequential cutoff working. A partial hit counts as a miss, and above 95 percent cache use everything bypasses. Congestion thresholds of 2 ms for reads and 20 ms for writes make bcache stand down when the cache device is slow, so a cache that stops caching under load has usually not failed.
dm-writecache: the one that does not cache reads
The kernel documentation: “It doesn’t cache reads because reads are supposed to be cached in page cache in normal RAM.” No read ever pulls a block into this cache. It makes writes return faster, so it suits synchronous writers - a database, a mail spool, an NFS server - and does nothing for find runs or VM boot storms.
Then the figure that decides most designs. lvmcache(7): at 4096-byte blocks, “each 100 GiB of writecache cachevol uses slightly over 2 GiB of system memory”; at 512-byte blocks, “a little over 16 GiB”. Scaled to a terabyte:
1 TiB cachevol, block_size=4096 -> about 21 GiB of RAM
1 TiB cachevol, block_size=512 -> about 164 GiB of RAM
Size the writecache against your RAM, not against your NVMe. Match block_size to the filesystem, though: an XFS made with 512-byte sectors (check xfs_info for sectsz=) will likely fail to mount behind a 4096 writecache. Its exposure window is autocommit_blocks, 65536 on SSDs, and autocommit_time, 1000 ms. Attach it with lvconvert --type writecache --cachevol fast --cachesettings 'block_size=4096' vg/main; to detach, set cleaner=1, wait for the writeback count to reach zero, then lvconvert --splitcache. Every write it holds exists on one device, so here a cache drive without power-loss protection is a defect, not a trade-off.
bcache or LVM cache
| bcache | LVM cache (dm-cache) | |
|---|---|---|
| Retrofit onto live data | no, format required | yes, lvconvert in place |
| Mirrored cache device | only via md underneath |
documented, --type raid1 |
| Small random write wear | lower, log-structured | chunk-sized promotions |
| Sequential bypass | yes, sequential_cutoff |
no tunable on smq kernels |
| Load-sensitive throttling | yes, congestion thresholds | no equivalent |
| Tunables that still work | many, documented | few; smq ignores most |
| Default write mode | writethrough | writethrough under LVM |
Choose bcache for a volume built from scratch with small random writes; choose LVM cache when the data already exists or the cache must be mirrored without a second layer. No vendor-neutral measurement says either is faster.
btrfs, bcachefs, Open CAS and the dead ends
btrfs has no cache of its own, so you build one of the three underneath. On a multi-disk btrfs raid1, cache each drive and mirror above it: mirroring below with md and caching once leaves btrfs one device, and a checksum failure becomes an I/O error instead of a repair from the second copy. bcachefs has the cleanest tiering design on Linux, but it left mainline with Linux 6.18, released November 30, 2025, and ships out of tree. Set durability=0 on its cache devices, or new writes can sit only on the flash until they are moved. Open CAS is an out-of-tree module that trails the kernel - release 26.03.4, published August 24, 2026, supports kernels up to 7.0 - and needs about 21.6 GB of RAM per 1 TB of cache at its default line size. flashcache and EnhanceIO, still recommended in old forum threads, were never merged. For most readers, LVM cache is the answer.
Windows: Storage Spaces tiering, and what is left
Storage Spaces tiering is not a cache. It moves data on a timer, and the timer is the whole story. Access counts are recorded through the day, and a task moves sub-file extents between the tiers at 1:00 a.m. A working set that changes at 9:00 a.m. is served from spinning disks until the next night, so tiering suits a footprint that is stable for days, such as a VM parent image. You buy as many fast devices as slow ones, so it is a poor fit for one NVMe in front of two hard drives; in return, a dead SSD is an ordinary failure the mirror handles. Windows Server 2022 also has a storage bus cache for a standalone server with Failover Clustering installed; Microsoft lists read-and-write caching only for a simple space with no resiliency, and read caching for mirror-accelerated parity, whose mirror tier still caches writes in the default Shared provision mode.
On the desktop, Intel’s consumer answers are gone: Smart Response Technology stopped at Intel RST 15.9, and Optane Memory support was dropped from RST 19.0 onward, so a second-hand Optane Memory module no longer gets you Windows caching, only an ordinary NVMe device that bcache or dm-cache will accept. If you inherit a live SRT volume, leave the driver alone: once it is disabled under RST 16.0 or later it cannot be re-enabled. ReadyBoost, a read cache on USB flash, is no answer either. That leaves PrimoCache from Romex Software, with a RAM cache at level 1 and an SSD or NVMe cache at level 2 that persists across restarts. Its Defer-Write holds writes back for a set number of seconds, and Romex recommends it only for “temporary, unimportant, or reproducible data”: deferred writes can land out of order, so a power cut leaves a mix of old and new blocks rather than an older consistent state. RAM caching covers that trade in full.
NAS appliances: Synology, QNAP, unRAID, TrueNAS
On an appliance you do not choose a caching architecture; you choose a vendor, and the architecture comes with it.
Synology builds a read-only cache from one SSD, and a read-write cache that “must be created with at least two SSDs of the same type (i.e., NVMe or SATA)”: the product refusing to put acknowledged writes on one device. Its sizing advice is honest: for 100 GB of hot data, growing the cache from 100 GB to 500 GB “will not result in significant performance increase.” The SSD Cache Advisor measures the hot set over at least 7 days. RAM rarely limits it:
Synology mapping table: ~400 KB of RAM per 1 GB of SSD cache
1 TB of cache: 1024 x 400 KB = 409,600 KB ~ 400 MB
DSM spends at most 25% of installed RAM on that table, so
1 GB of RAM allows: 1,048,576 KB / 400 KB per GB ~ 2,600 GB of cache
a 4 GB NAS allows: (4 GB / 4) = 1 GB of table -> ~2.6 TB of cache
On Synology, RAM almost never limits the cache. On ZFS it usually does. Alpine-CPU models are the exception, capped at 930 GB of cache. “Pin all metadata” keeps the Btrfs metadata on the cache, a special vdev without the permanence. And Synology says “SSD cache will not improve performance in scenarios involving sequential access patterns”, naming single-channel HD video streaming and surveillance recording.
QNAP ships a cache and a tier. SSD cache copies data as it is accessed; Qtier “writes incoming data to the SSD tier and moves data to different tiers based on access frequency”, so Qtier SSDs hold the only copy of what lives there and need their own redundancy. SSD cache is capped by memory: QTS 5.0.0 and later list 2 GB of RAM for 1 TB of cache and 8 GB for the maximum, 8 TB on ARM models and 16 TB on x86.
unRAID’s cache pool is not a cache. New files land on the pool and the mover shifts them to the array on a schedule; reading a file already on the array never promotes it. Array parity does not cover pools, so a single-device pool holds new files in one place until the mover runs.
TrueNAS is ZFS with a web interface, so the ZFS sections apply unchanged; a special vdev on a raidz pool is just as permanent there. Releases before 24.04 shipped with persistent L2ARC turned off.
Before buying, ask whether the cache can be removed without destroying the array. Synology and QNAP SSD caches attach to an existing volume and detach again, and Qtier can be enabled in place. Then ask what it costs in bays: a read-write Synology cache takes two M.2 slots or two drive bays for good. NVMe in M.2 shows what physically fits.
Choosing the NVMe: write budget first, then power-loss protection
The cache device’s wear has little to do with how much the workload writes. Every promotion is a write, so a read-only job can push terabytes a day onto the flash: a 4 KiB read miss writes 16 times its size at LVM’s 64 KiB default chunk, 256 times at 1 MiB. Mirroring doubles it.
Promotions accepted by the policy 25 /s (measure: the
promotions counter
in dmsetup status)
Chunk size from lvs -o+chunksize 1 MiB
Bytes written to flash per second 25 MiB/s = 26.2 MB/s
Per day 26.2 MB/s x 86400 = 2.26 TB
Samsung 990 Pro 2 TB, 1200 TBW rating:
1200 TB / 2.26 TB/day = 531 days ~ 17 months
Solidigm D7-P5810 800 GB, 50 DWPD x 5 y = 73 PB:
73000 TB / 2.26 TB/day = 32301 days ~ 88 years
Twenty-five promotions a second is a quiet cache. The consumer drive in that example is the Samsung 990 Pro 2 TB; TBW, DWPD and how long an SSD lasts covers what its rating promises.
Power-loss protection is the second filter. With capacitors, a drive can complete a FLUSH once the data is in its own DRAM, which is why protected drives answer fsync in microseconds and consumer ones in milliseconds. And FLUSH honesty cannot be read off a datasheet: a 2022 power-pull test of four consumer NVMe drives found two, a Sabrent Rocket and an SK hynix Gold P31, losing data they had already flushed, and the FAST ’13 study by Ohio State and HP Labs found 13 of 15 SSDs failing in ways their interface said they would not.
A used enterprise drive is usually a better buy than a new consumer one at the same price. A drive that has spent 20 percent of its rated life on a 50 DWPD budget has more writes left than a new 0.33 DWPD consumer drive has in total. Start from the enterprise SSD listing, whose class marks duty rating, and confirm PLP on the exact model’s datasheet. The endurance rating goes with the model line rather than the capacity: Micron labels its 7450 MAX “Mixed-Use, 3 Drive Writes per Day”, 400 and 800 GB M.2 models included (product brief, Rev. C, February 2026), while Kioxia’s PM6-M, a SAS SSD, is 10 DWPD from 400 GB to 3.2 TB (product brief, Rev. 1.2, 2023). Enterprise drives explains the classes, and buying used drives on eBay covers checking a pulled drive’s wear. Leave 10 to 20 percent of it unpartitioned so its own write amplification falls, and watch the wear from day one:
nvme smart-log /dev/nvme0 -H
data_units_written counts units of 1000 x 512 bytes, so multiply by 512,000 for bytes. Record it weekly; the slope gives the replacement date. percentage_used is a vendor estimate that can pass 100 percent without the drive failing, so also watch available_spare and critical_warning bit 3, which means the media has gone read-only. Budget for the replacement at build time: the cache device is a wear part.
Checking that it worked
Every stack ships counters; none graphs them for you. On LVM cache, read the cache_read_hits and cache_read_misses fields of lvs, as in recipe 1; they are 32-bit and wrap, so graph differences, never totals. On bcache, read the decaying windows, with bypassed beside the ratio:
cat /sys/block/bcache0/bcache/stats_hour/cache_hit_ratio
cat /sys/block/bcache0/bcache/stats_five_minute/bypassed
On ZFS, check the ARC hit ratio first, since anything that reaches L2ARC is something the ARC missed, then read the kstat file rather than a tool’s formatted output:
awk '/^l2_(hits|misses|size|hdr_size)/ {print $1, $3}' /proc/spl/kstat/zfs/arcstats
On Windows tiering, the Storage Tier Optimization Report from defrag D: /g /h /# is the measurement.
Read the ratio against the model above: 80 percent still leaves one request in five at 12 ms. Then run iostat -x 1 on the hard drive, the NVMe and the combined device at once: a healthy cache shows low r/s on the hard drive and a larger rareq-sz there, because what reaches it is now migrations and destages.
A useful synthetic test has a file larger than the cache, a skewed distribution and a discarded warm-up:
fio --name=hotset --filename=/mnt/cached/testfile \
--size=2T --io_size=200G \
--rw=randread --bs=4k --iodepth=32 --numjobs=4 \
--ioengine=io_uring --direct=1 \
--random_distribution=zipf:1.2 --norandommap --randrepeat=0 \
--ramp_time=300 --runtime=1800 --time_based \
--percentile_list=50:90:99:99.9:99.99 --group_reporting
Report clat percentiles, not bandwidth. A sequential benchmark will show the cache doing nothing, and that is the cache behaving correctly. Every first measurement runs cold, because caches start empty. Prove the cache earned its place by difference: counters, nvme smart-log and p99 before, a week of real work, then all three again. If the hit ratio is 60 percent and percentage_used moved two points in a week, you bought wear.
Power cuts, dead drives and removal
Losing a read cache costs you time. Losing a writeback cache costs you data. Losing a tier costs you the data that lived only there. And “no data loss” is not “keeps working”: under a writethrough dm-cache, a dead cache device returns errors on every read hit until you detach it. That is a reading of the kernel source, not a documented guarantee, so rehearse the detach.
| Stack | Mirrored cache | How |
|---|---|---|
| LVM dm-cache | Yes | lvcreate --type raid1 -m 1 for the fast LV, documented in lvmcache(7) |
| LVM dm-writecache | Yes | the same raid1 cachevol |
| bcache | Not natively | an md RAID1 underneath: common practice, not documented policy |
| ZFS L2ARC | No, and unnecessary | “Cache devices cannot be mirrored”; a read error is reissued to the pool |
| ZFS SLOG | Yes | log vdevs can be mirrored; raidz is not supported for them |
| ZFS special vdev | Mandatory | it holds the only copy, so it matches the pool’s redundancy; if one member fails, replace it now; losing the whole vdev loses the pool |
Two drives of the same model, bought together and written identically, wear out together; stagger the pair or replace one early.
After a crash, dm-cache is the sharp one. The dirty bit is a hint, and “if the system crashes all cache blocks will be assumed dirty when restarted”, so every block must be written back before the cache can be detached:
1 TiB writeback cache, all blocks presumed dirty after a crash
HDD sustained sequential write (IronWolf Pro 20 TB spec): 285 MB/s
1,099,511,627,776 B / 285,000,000 B/s = 3,858 s = 64 min
That is a floor, because a destage is not one long sequential write. bcache “simply doesn’t return writes as completed until they’re on stable storage”, so a crash is a handled case and a missing cache device is the dangerous one. ZFS has no writeback block cache to recover. Pulling a cache device without detaching it is how people lose pools: each recipe above ends with its removal steps, and a dead, unmirrored writeback cache holding dirty data means restoring from backup.
If the server has a hardware RAID card
A used server often arrives with a RAID card carrying its own DRAM write cache behind a battery or supercapacitor. Some carry none: Broadcom’s MegaRAID 9440-8i has no cache, while the 9460-8i has 2 GB of DDR4-2133 and the 9460-16i 4 GB. A dead battery or capacitor drops the array to write-through, and on RAID 5 or 6 small random writes collapse to disk latency. The forum fix, Always Write Back, keeps write-back on with no working energy backup; replace the module instead. Under ZFS, run the card in HBA mode, which drive interfaces covers.
A cache is not a backup, and a mirror is not a backup either. A mirrored writeback cache survives one dead NVMe, not a deleted file.
When caching is the wrong answer
The working set is larger than any cache you would buy. With no hot region, the hit ratio is roughly cache size over footprint:
h ~= C / F uniform random reads, cache C over footprint F
F = 6 TB, C = 500 GB -> h = 0.083
L_eff = 0.083 * 0.1 ms + 0.917 * 12 ms = 11.0 ms
Speedup = 12 / 11.0 = 1.09x
A 9 percent speedup, and every miss a read-promoting cache promotes costs a chunk-sized write to the NVMe: endurance spent on a rounding error of latency.
The workload is sequential. At the 285 MB/s an IronWolf Pro 20 TB is rated for, streaming media or a restore is not disk-bound. bcache’s 4 MiB cutoff, ZFS’s l2arc_noprefetch and Microsoft’s advice to pin media VHDs to the HDD tier all exist to keep that traffic out of the cache.
The money belongs in RAM. A read the page cache already served is not a problem flash can fix, and an L2ARC makes RAM scarcer.
The hot data fits on an SSD outright. If you know which 800 GB is hot - the database, the VM images, the build tree - put it on an SSD: a hit ratio of 1.0 by construction, no promotion writes, no drain after a crash. Compare the price per terabyte across every drive we list with an SSD big enough for the working set, watch both on Trends, and see hard drive or SSD for the split, or NVMe arrays if it takes several drives.
The filesystem or the drive was at fault. Turn off access-time updates first, unless something reads them, such as a Maildir client: under Linux’s default relatime, a nightly backup that reads a few million files makes a few million metadata writes, each a dirty block in a writeback cache, and noatime (atime=off on ZFS) makes it zero. A ZFS media pool left at the default 128 KiB recordsize carries eight times the block count, and so eight times the L2ARC header cost and special-vdev metadata, of the same bytes at 1 MiB. And if the drive underneath is shingled, a write cache postpones the rewrite penalty and never removes it.
If your problem needs no read cache, write cache or tier, the cheapest correct configuration is the one you already have, with more RAM in it.