In this guide: 14 sections
- The short answer, by drive count
- Do this, and avoid this
- Why a mirror beats a faster stripe
- PCIe lanes are the budget
- Bifurcation, switch cards and cables
- What physically fits: M.2, U.2, U.3 and EDSFF
- Home builds, from two M.2 slots to a used server
- Skip hardware RAID for NVMe
- Linux md on flash: the defaults to change
- ZFS on NVMe: mirrors, ashift and recordsize
- btrfs, XFS, ext4 and the rest
- Endurance, power-loss protection and heat
- Failure and rebuild on flash
- Buying the drives
For most home builds, the right NVMe array is two drives in a mirror, in M.2 slots wired to the CPU, under ZFS or btrfs so every read is checksummed. Count the PCIe lanes before you count the drives: each NVMe drive is an x4 link, a current consumer board leaves about 24 usable CPU lanes, and extra M.2 slots usually share one narrow link to the chipset. Past two drives the motherboard manual decides what works, because a passive quad-M.2 card needs a slot that bifurcates; past six, a used EPYC 7002 server or Dell Precision 7920 workstation is where lanes stop being scarce. Parity, hardware RAID and cache tiers buy complexity first and a benefit second, and a mirror is not a backup.
The short answer, by drive count
An NVMe array fails at the platform long before it fails at the drive, so the first decision is how the drives attach, and it follows from how many there are:
Drives Board bifurcates? Do this
------- -------------------- ------------------------------------------
1 to 2 irrelevant CPU-attached M.2 sockets, nothing else
3 to 4 yes, x4x4x4x4 passive carrier in the CPU x16 slot
3 to 4 no switched carrier, or single-drive risers
5 to 8 yes, lanes to spare two bifurcated slots, watch the chipset
5 to 8 no spare lanes switched card, accept the uplink as the cap
9 to 24 either server board, U.2 or E3.S backplane, switch
25+ either a chassis decision, not a card decision
Bifurcation first, because it is free. A switch card when the board will not split, priced against what it adds in heat and oversubscription. A server chassis when the drive count passes what either can carry, at which point the backplane, not the card, is what you are buying. The rows are argued in bifurcation, switch cards and cables.
Four tiers of build
| Tier | Drives | Lanes reaching the drives | Thermal arrangement | Filesystem that fits |
|---|---|---|---|---|
| Mini-PC, 2-slot NAS | 2 x M.2 | 8, CPU-attached | still air, 3.3 V only | ZFS mirror or btrfs raid1 |
| Desktop tower | 4 to 6 x M.2 | 24 CPU lanes, or 4:1 behind the chipset | carrier plus a slot fan | mirrors, or raidz1 with the endurance cost counted |
| Used workstation or server (Precision 7920, EPYC 7002) | 8 to 16, M.2 on quad cards or U.2 | 96 Gen3 CPU lanes on two Xeon Scalable, 128 Gen4 on one EPYC 7002; lanes no longer the constraint | hot-swap cage and fan wall (server), actively cooled quad cards (7920) | ZFS mirrors or raidz2 |
| Past the ceiling | 24 and up | 96 needed, or a switch at 6:1 | designed airflow, rack noise | any of them; the CPU is the limit now |
Each tier is worked through in home builds, from two M.2 slots to a used server.
Which redundancy and which filesystem
- Two drives: a ZFS mirror or a btrfs
raid1. Both verify every block they read and repair from the good copy; md RAID1 under ext4 or XFS cannot tell which copy is right. - Four to eight drives: ZFS mirrors striped together for random I/O and fast rebuilds, or raidz2 for capacity with the endurance cost counted.
- Small random overwrites on data you can rebuild: md RAID10 under XFS, with no parity and no padding.
- Avoid: btrfs RAID5 and RAID6, a hardware RAID card in front of NVMe, and any special vdev, journal or log device on a single drive.
The figures below come from named specifications, datasheets, kernel documentation and dated trade-press reports, with the arithmetic shown and illustrative values labeled as such.
Do this, and avoid this
Do:
- Count the lanes before you count the drives. On AM5, a graphics card at x16 leaves 8 CPU lanes, which is two drives; at x8 it leaves four drives.
- Read the board manual before buying a passive quad-M.2 card. CPU support, board routing and a firmware option must all be present, or only the drive on lanes 0 to 3 appears.
- Prefer mirrors. A partial-stripe write to RAID 5 costs two reads and two writes, RAID 6 three of each; a mirror costs one write per copy and no read.
- Set ashift and recordsize before the first byte lands. ashift is fixed for the life of the vdev, and a recordsize change applies only to data written after it.
- Raise md’s rebuild limit.
speed_limit_maxships at 200000 KiB/s per device, which runs a flash rebuild at hard-drive pace. - Enable autotrim or fstrim, and confirm it reaches the drives. md refuses discard on RAID 4, 5 and 6 by default.
- Put moving air over every M.2 module. A throttling member holds up every write in a mirror.
- Split the purchase across two sellers or batches, and read Percentage Used, Available Spare and Data Units Written before paying.
- Scrub on a schedule and read the number it returns.
- Keep a copy outside the array, and restore from it once.
Don’t:
- Don’t use btrfs RAID5 or RAID6 for data you care about; the project marks them unstable.
- Don’t put a drive without power-loss protection under a synchronous workload. Each flush waits on NAND.
- Don’t fill a pool past about 80 percent.
- Don’t read the box figure as sustained. A consumer drive’s headline write speed is an SLC cache number that ends when the cache does.
- Don’t trust hot-plug on M.2. U.2, U.3 and the EDSFF bays are designed for it; the M.2 socket is not.
- Don’t call redundancy a backup. A mirror replicates a mistaken deletion in microseconds.
Why a mirror beats a faster stripe
Most NVMe arrays that disappoint were built with SATA habits: a controller card in a spare slot, a stripe across every drive present, a chunk size copied from an old forum thread. Those habits fit a device that sat behind a host bus adapter and delivered 550 MB/s. An NVMe drive is a PCIe endpoint the CPU addresses directly, with up to 65,535 I/O queues where AHCI had one 32-slot command list per port, so each core can own a queue. And one drive now outruns the array it replaced:
Four SATA SSDs striped, best case 4 x 550 MB/s = 2.2 GB/s
One Gen4 x4 NVMe SSD, datasheet about 7.0 to 7.4 GB/s
One Gen5 x4 NVMe SSD, datasheet about 14 to 14.9 GB/s
7.2 / 2.2 = 3.3x 14 / 2.2 = 6.4x
Both NVMe rows are vendor datasheet claims for large sequential reads, and random work looks nothing like them. Even so, one drive outruns four striped SATA SSDs by a factor of three to six, and the limits it meets are on the host side of the connector. Four Gen5 drives reading flat out move 56 GB/s of DMA into memory, more than half the 96 GB/s theoretical peak of dual-channel DDR5-6000, so channel count is worth checking before drive count (DDR5 modules and rates).
Striping was how a SATA build got bandwidth. That argument has expired: the network in front of a home server is 1, 2.5 or 10 Gb/s, and 10 Gb/s Ethernet asks for 1.25 GB/s, under a fifth of one drive. A second drive adds no bandwidth a single drive was failing to deliver. What it adds is that the array survives one of them dying. That is why a two-drive mirror is the whole of the right answer for most home builds: no parity arithmetic, no journal device, no write hole and no chunk-size decision. The shopping list is two drives and a second CPU-attached slot, and the NVMe listings sort by price per terabyte by default (M.2 only).
PCIe lanes are the budget
The drives are almost never the constraint. The board is. A current PCIe 4.0 x4 NVMe SSD reads at something near 7 GB/s. Four of them in four M.2 sockets on a consumer board do not read at 28 GB/s, because three of those sockets usually sit behind a single link a third as wide as the demand they present to it.
What one drive asks for
x4 is the norm: the M.2 M-key socket defines PCIe x4, and the U.2 connector gives the device up to four lanes. x2 devices exist and are sold without saying so. A module notched B+M fits either socket but has only two PCIe lanes on the B-key pins, so it is capped at x2 whatever it is plugged into, and some controllers implement only two lanes in an ordinary M-key package.
The trained link is the honest source. lspci -vv prints LnkCap, what the
device can do, and LnkSta, what it actually negotiated. A drive reporting
“Speed 8GT/s, Width x2” is on a Gen3 x2 link at roughly 1.97 GB/s regardless of
what the seller wrote, and that is the check to run on arrival.
Lanes per generation
PCIe 1.0 and 2.0 lose 20 percent of the raw rate to 8b/10b line coding. From Gen3 onward every 130 bits on the wire carry 128 of payload, so a Gen3 lane carries 8.0 GT/s x 128/130 = 7.877 Gb/s, or 0.985 GB/s per direction:
| Generation | Rate per lane | Line code | Payload per lane | At x4 |
|---|---|---|---|---|
| PCIe 1.0 | 2.5 GT/s | 8b/10b | 0.250 GB/s | 1.000 GB/s |
| PCIe 2.0 | 5.0 GT/s | 8b/10b | 0.500 GB/s | 2.000 GB/s |
| PCIe 3.0 | 8.0 GT/s | 128b/130b | 0.985 GB/s | 3.938 GB/s |
| PCIe 4.0 | 16.0 GT/s | 128b/130b | 1.969 GB/s | 7.877 GB/s |
| PCIe 5.0 | 32.0 GT/s | 128b/130b | 3.938 GB/s | 15.754 GB/s |
| PCIe 6.0 | 64.0 GT/s | PAM4, FLIT with FEC | 7.563 GB/s | 30.25 GB/s |
The Gen6 row is the link after FLIT framing, counting the CRC and FEC bytes as overhead. Every figure is a ceiling before packet headers and flow control take their share, so a Gen4 x4 drive rated at 7.4 GB/s sequential read is running at about 94 percent of its 7.877 GB/s ceiling. The first Gen6 SSDs that trade press reports in mass production in 2026, Micron’s 9650 and Samsung’s PM1763, are enterprise parts: for anything you can build at home, Gen5 is the ceiling, and the used platforms below are Gen3 and Gen4.
Lane budgets by platform, new and used
- Consumer desktop, about 24 usable CPU lanes. AMD’s AM5 has 28 PCIe 5.0 lanes, four of them spent on a chipset link that runs at Gen4; Intel’s LGA1851 has 20 PCIe 5.0 plus 4 PCIe 4.0. The conventional split is x16 to graphics, x4 to one M.2 socket, x4 left over.
- New workstation, roughly 48 to 128. Threadripper PRO 9000WX on WRX90 provides 128 PCIe 5.0 lanes and Xeon W-3500 up to 112.
- New server, roughly 88 to 128 per socket. EPYC 9005 provides 128 per socket, and Xeon 6 provides 88 on the 6700P and 96 on the 6900P.
- Used server, 128 at Gen4 on one socket. AMD lists the EPYC 7002 series at “128 PCIe 4.0 Lanes”, and a two-socket board still has about 128 in total, as the used EPYC section explains.
- Used workstation, 48 per CPU at Gen3. Intel’s specification pages for 1st and 2nd Gen Xeon Scalable parts, the Xeon Gold 6130 and Gold 6230 among them, list 48 PCIe 3.0 lanes per processor, so two sockets give 96.
A 24-bay U.2 chassis wants 96 lanes of drive attachment before a network card gets one, which explains the shape of every dense NVMe server: a high-lane socket, two sockets, or a PCIe switch. Memory for these platforms is on ram-index’s processor pages.
The chipset uplink, and the manual
Extra M.2 sockets almost always hang off the chipset, and the chipset is a switch with a fixed uplink. AM5’s Promontory reaches the CPU over one PCIe 4.0 x4 link, and a board with a second chipset die daisy-chains it behind the first; Intel’s DMI 4.0 link is x8 on Z890 and x4 on B860 and H810. Four chipset M.2 sockets present 16 lanes of demand to an uplink four to eight lanes wide, shared with USB, SATA and the onboard network. That is the wrong shape for an array: a rebuild, a scrub or a nightly backup asks every member for sequential throughput at the same moment, which is exactly the case an uplink cannot serve. A chipset-attached array can look fine in a desktop workload and then take a day to rebuild.
Boards also borrow. Populating the second M.2 socket may disable two SATA ports, drop the primary x16 slot to x8, or force a third socket to Gen3 x2, and the same CPU on a different board shares differently. The block diagram and the slot-configuration table in the motherboard manual are the authority; read them before ordering drives. What plugs into what covers the connector half of the same question.
Worked example: how many drives can this board feed at full rate?
Board: consumer AM5, 24 usable CPU lanes, chipset on PCIe 4.0 x4
Plan: eight PCIe 4.0 x4 NVMe drives, headless, no graphics card
Given: CPU, board and firmware all split the x16 slot x4/x4/x4/x4.
Plenty of consumer boards do not, which ends the exercise at
two drives fed at full rate. Check the manual, not the chipset.
CPU-attached
x16 slot bifurcated x4/x4/x4/x4 ..... 4 drives at 7.877 GB/s each
CPU M.2 socket, x4 .................. 1 drive at 7.877 GB/s
lanes spent ......................... 20 of 24
Chipset-attached
3 further M.2 sockets, x4 each ...... 23.63 GB/s of demand
uplink, PCIe 4.0 x4 ................. 7.877 GB/s of supply
ratio ............................... 3:1, before USB, SATA and the NIC
Note the four spare lanes: on AM5 they are the second CPU-attached
x4 group, and a board that exposes them as an M.2 socket feeds a
sixth drive at full rate. This example assumes it does not.
Answer
drives attached and working ......... 8
drives fed at full rate, together ... 5
aggregate with all eight busy ....... 5 x 7.877 + 7.877 = 47.3 GB/s
the sum of eight x4 ceilings ........ 8 x 7.877 = 63.0 GB/s
No filesystem or RAID level recovers the gap between those last two lines. Count the lanes first, then count the drives. The same eight drives on a 128-lane socket are eight independent x4 links with 96 lanes still unspent, and the arithmetic stops mattering, which is what makes a used server worth a look.
Bifurcation, switch cards and cables
The cheapest four-drive NVMe card is the commonest wasted purchase in this subject. A passive quad-M.2 carrier is wiring; it contains no PCIe device at all. Plugged into a board that cannot split its x16 slot, exactly one drive appears - the one on lanes 0 to 3 - and the other three are invisible, with no error message anywhere to say why.
Bifurcation is a firmware setting
Bifurcation splits one lane group into several independent links, each with its own link training. Nothing is routed and no lanes are created: an x16 slot set to x4x4x4x4 is four x4 links. The split is applied at power-on, so it lives in the BIOS or UEFI and needs a reboot, and three things must agree: the CPU must support it, the board must route the lanes to that slot, and the firmware must expose the option. Intel server firmware shows it per IIO stack, labeled IOU0/IOU1/IOU2 with values such as x4x4x4x4 and x8x8; AMD platforms show a per-slot “PCIe lane bifurcation” item. If neither the board manual nor the BIOS menu names the slot and a split, assume the board cannot do it. No card corrects that, which is why the manual comes before the third and fourth M.2 drives.
Switch cards add endpoints, not bandwidth
A carrier with an onboard PCIe switch needs no bifurcation, because the switch is a real device with one upstream port and several downstream ones, and it works on any host. It costs what the switch costs, adds tens of watts of heat that needs directed airflow, and adds a latency hop vendors describe as “cut-through” without publishing a figure. It does not create bandwidth: twenty-four drives at x4 behind an x16 uplink is 6:1 oversubscription, fine for a capacity array where no two clients read at once and poor for a database that wants every drive at once.
Cables, retimers and Gen5
Past an M.2 socket, drives connect by cable: SlimSAS (SFF-8654) is the internal cable the Gen4 era settled on, and MCIO (SFF-TA-1016) is what Gen5 designs use. A cable is a signal-integrity component from Gen4 upward. Astera Labs puts the Gen5 insertion loss budget at 36 dB at 16 GHz, with roughly 10 to 20 percent held back as design margin by rule of thumb, and both packages, the boards, every connector and the cable share it. Longer Gen5 runs need a retimer, and a link may have at most two retimers, which is why riser plus cable plus backplane is a topology Gen5 refuses. Buying used Gen3 or Gen4 U.2 sidesteps the whole problem.
What physically fits: M.2, U.2, U.3 and EDSFF
The cheapest NVMe on the used market is in the form factor your machine has no socket for. Decommissioned U.2 enterprise SSDs sell cheaply per terabyte because nearly every consumer board ships M.2 sockets and no 2.5-inch NVMe bay. The connector detail lives in SATA, SAS, NVMe and M.2: what plugs into what, and where the cheap U.2 stock comes from in enterprise drive pulls. For an array, five facts decide what fits:
- M.2: check length, key and sides. 2280 is the default length; 22110 is the server length, and the extra 30 mm is usually power-loss-protection capacitors. An M-key socket can be SATA-only, and an NVMe drive in one does not appear at all. A double-sided module may not seat in a thin bay.
- M.2 is 3.3 V only, screw-retained and not designed for hot-plug. SNIA’s EDSFF explainer says M.2 110 mm had “challenges in terms of hotplug/serviceability, thermals and overheating” in hyperscale data centers.
- U.2 wants 12 V and a bay. It is a 2.5-inch drive on the SFF-8639 connector, usually 15 mm thick and rated to 25 W, with staged mating for hot-plug. Without a backplane, attach it through a host adapter card with OCuLink, MiniSAS HD, SlimSAS or MCIO cables, or an M.2-to-U.2 adapter with its own power feed. Listings: NVMe in U.2.
- U.3 compatibility runs one way. SFF-TA-1001 mandates that U.3 drives work in U.2 bays, and its interoperation table lists a U.2 drive in a U.3 bay as “No link”. Some backplanes are wired dual-mode anyway, so check the vendor’s documentation for the exact part.
- EDSFF needs its chassis. E1.S, E1.L, E3.S and E3.L have power envelopes from 12 W to 70 W and defined airflow; buy them only with the chassis.
Hot-plug also needs the slot to advertise it in firmware and the operating system to be given native control, and surprise-removal races were still drawing unmerged patches on the linux-pci list in September 2026. Treat “surprise removal works” as a claim about one kernel on one platform.
Home builds, from two M.2 slots to a used server
An array buys availability and nothing else. It does not buy capacity or durability, and a second copy on the same power supply behind the same filesystem is not a backup. Each tier is worth building once you can say why the data has to keep serving through a drive failure.
Two M.2 slots, and why most people should stop there
The mini-PC or the two-slot M.2 NAS. Both sockets usually hang off the CPU, and the recipe is a mirror:
2 x M.2 Socket 3, CPU-attached, Gen4 x4 each:
mirror read ~ 2 x 7.877 = 15.75 GB/s (md picks the member with fewer
pending requests once any device
reports non-rotational)
mirror write = 7.877 GB/s (both copies are written)
10GbE client = 1.25 GB/s
2.5GbE client = 0.3125 GB/s
The read figure is a ceiling: a single sequential reader sees roughly one drive. One drive’s x4 link already outruns 10 gigabit Ethernet by a factor of six, so anything above two drives at this tier is spent on a link that cannot carry it. In a fanless mini-PC the two modules often sit with no defined airflow, and throttling arrives as a collapse in sustained write rate rather than as an error anyone logs.
The cost: consumer M.2 prices run close enough to linear that two 2 TB drives often come to about what one 4 TB drive costs, and leave you with 2 TB. That is the full price of double the capacity, spent on uptime. Rank the candidates by price per terabyte across every drive first, because that ratio moves week to week on Trends.
A desktop tower with four to six drives
AM5 and LGA1851 each leave 24 CPU lanes. On a board that exposes x4x4x4x4 on the x16 slot, drop the graphics card and six drives sit on the CPU with no sharing. The chipset M.2 slots are where builds quietly go wrong:
Four chipset M.2 slots, AM5 Promontory (Gen4 x4 uplink):
drive demand 4 x 7.877 GB/s = 31.5 GB/s
uplink supply 1 x 7.877 GB/s = 7.9 GB/s
ratio = 4:1, shared with USB, SATA and the NIC
On boards with two daisy-chained chipset dies, the ports on both still share that one link. Four chipset drives in a stripe are one drive wide.
A quad carrier parks four modules in the dead-air pocket behind the graphics card, and the inner drives set the throttle point for the whole array. Put it in the top slot, add a slot fan or a carrier with its own blower, and treat any card whose heatsink is a flat plate as a two-drive card.
Four to six drives is where parity tempts. On md RAID5 the write hole is real, and the fixes cost write speed or a power-loss-protected journal drive; ZFS raidz1 avoids the hole by construction. The second cost is endurance: a small md RAID5 write puts roughly twice the host bytes on the drives and RAID6 three times, straight off the rated DWPD, and RAIDZ pays in padding instead. Mirrors avoid both costs, at half the capacity.
A used EPYC 7002 server or Dell Precision 7920
Two used platforms show how far the arithmetic moves.
A single-socket EPYC 7002 board has 128 PCIe 4.0 lanes, eight DDR4 memory channels and up to 64 cores, by AMD’s own figures: 32 drives’ worth of x4 attachment at 7.877 GB/s each. One socket is the better buy for an array, because in a two-socket board each CPU gives up lanes to link to the other: the platform was designed for 128 lanes in total in a dual-socket system as well, ServeTheHome explained in April 2019, and reaching 160 meant routing socket-to-socket links to drives instead, as Dell’s PowerEdge R7525 did. One trap is specific to the used market: AMD’s footnote says a 7002 on a board designed for the first-generation EPYC needs a BIOS update, that PCIe 4 support is a question for the server maker, and that “a motherboard designed for 2nd Gen EPYC processors is required to enable all available functionality”. Buy the board for Gen4, not just the socket.
The Dell Precision 7920 tower takes one or two 1st or 2nd Gen Xeon Scalable processors at 48 PCIe 3.0 lanes each, and every slot is PCIe Gen 3: two x16 slots with one CPU, two more x16 with the second, plus narrower x8, x4 and x1 slots. Dell’s specification lists up to eight M.2 NVMe drives on two Dell Ultra-Speed Drive Quad x16 cards, “maximum requires dual CPUs”, and up to four front hot-plug M.2 or U.2 NVMe drives in FlexBays on chassis with the PCIe backplane, more than two of them only with dual CPUs, as StorageReview’s spec table of March 2020 records. Eight drives at Gen3 x4 is 8 x 3.938 = 31.5 GB/s, still about 25 times a 10 GbE link, so the older generation costs a home array nothing it can use. Its built-in NVMe RAID is Intel VROC, RAID 0, 1 and 10 on the M.2 drives of the first Ultra-Speed Drive card, behind a motherboard activation key; skip hardware RAID for NVMe explains why a native md array, which needs no key and no VMD setting, is the better choice. ram-index lists memory for Dell’s PowerEdge servers on its Dell compatibility pages, and Dell’s own specification gives the memory a Precision tower takes.
At this tier U.2 becomes the cheap option, bringing 12 V and power-loss protection as a fitted norm: NVMe in U.2, or two common enterprise sizes, 1.92 TB and 3.84 TB, with the endurance rating taken from the model’s datasheet, not the capacity. Enterprise U.2 assumes forced air, a hot-swap cage with a fan wall behind it, and the same drive in a still desktop case will throttle. Use ZFS mirrors for rebuild speed and IOPS or raidz2 for capacity, and leave the bulk capacity on spinning disk, for the reasons in where each still wins.
Where a home build ends
A 24-bay chassis needs 96 lanes of drive attachment, and past roughly eight to twelve drives the bottleneck moves to the host CPU: md parity handling, ZFS checksum and compression work, the interrupt load. A home build ends where the spare parts begin: a dedicated circuit, noise that needs its own room, a spare chassis on the shelf, a written rebuild procedure. Clusters such as Ceph buy availability across machines, not latency; a three-node cluster on NVMe has worse single-threaded write latency than the mirrored pair it replaced.
Skip hardware RAID for NVMe
The RAID-on-chip in a used server was sized for a shelf of SAS drives, and eight NVMe drives ask it for roughly eight times what it was built to move. Twelve SAS-3 SSDs at 1,200 MB/s each total 14.4 GB/s; one PCIe 5.0 x4 NVMe SSD reads about 14 GB/s on its own. The host link is the harder wall:
Gen4 x16 host link 16 x 1.969 GB/s = 31.5 GB/s
Gen5 x16 host link 16 x 3.938 GB/s = 63.0 GB/s
Drives to fill a Gen5 x16 card 63.0 / 14 = 4.5
Direct x4 drives, 32 device lanes 32 / 4 = 8
The 9600 product brief for Broadcom’s tri-mode MegaRAID cards states 240 SAS or SATA drives against 32 NVMe, the 32 reached through PCIe switching in the backplane, which the drives then share. Its published ceilings:
| Broadcom 9600 series, MegaRAID | Published figure |
|---|---|
| JBOD path, 4K random read | 6M to 6.4M IOPS |
| RAID 0/1, 4K random read | 4.5M to 6.4M IOPS |
| RAID 5, 4K random write | 900K to 1.1M IOPS |
The parity figure is about one sixth of the JBOD figure through the same card, though Broadcom publishes no RAID 5 read number, so the ratio sets a random write against a random read. The deeper problem is architectural: NVMe’s queue model exists so each core can talk to the drive without an intermediary, and a RAID-on-chip puts one back, staging every byte through its memory and serializing every command through its firmware. A tri-mode card earns its slot in a mixed estate of SAS SSDs and flash behind one backplane, not as a way to give NVMe drives their own performance back.
Intel’s VROC is not a card: VMD, a CPU feature, supplies hot-plug, LED
management and error isolation, and on Linux VROC is md driven by mdadm with
Intel’s metadata. The unlicensed Pass-Thru default does no NVMe RAID; Standard
adds RAID 0/1/10 and Premium RAID 5. Toggling VMD in firmware changes how drives
enumerate, so never flip it under a live array. On June 8, 2026, StorageReview
reported that VROC stewardship moved from Intel to Graid Technology, with
UEFI-based licensing replacing hardware keys for new deployments, so date
anything you read about it. AMD’s RAIDXpert2 needs the out-of-tree rcraid
module on Linux, and a closed driver tied to particular kernels is a root
volume a kernel upgrade can leave unbootable. A native md superblock assembles
on any Linux machine with mdadm.
What hardware RAID still buys is a battery- or flash-backed write cache that coalesces partial-stripe writes into full-stripe writes; without the backup the correct policy is write-through. A boot volume the firmware can find and an enclosure stack the OEM already wired are the reasons that survive. When a used server is cheap, its controller is sometimes why: in front of SAS drives a 9600-class card is the right part, and in front of eight NVMe SSDs it is the component the array waits on.
Linux md on flash: the defaults to change
md is the right default for NVMe: the code is mature, the metadata is documented, no license key stands between you and your array, and any Linux kernel assembles it. The defaults were chosen for disks with a seek penalty.
Mirrors and RAID10 first
Once any member reports non-rotational, md sends each mirror read to the disk with fewer pending requests. RAID10 is a native personality that takes any device count, and it buys a write path with no parity arithmetic: no read-modify-write, no stripe cache, no write hole.
mdadm --create /dev/md0 --level=10 --raid-devices=8 --chunk=256K \
--write-zeroes /dev/nvme{0,1,2,3,4,5,6,7}n1
--write-zeroes (mdadm 4.3) zeroes every member so the initial resync becomes
unnecessary; mdadm’s page says the result behaves “as if –assume-clean was
specified”.
What a parity write costs
For a write that does not fill a stripe, md chooses between a read-modify-write (RMW) and a reconstruct write (RCW), whichever reads fewer devices:
RAID5, N devices, N-1 of them data:
full-stripe write 0 reads + N writes
RMW 2 reads (old data, old P)
+ 2 writes (new data, new P) = 4 operations
RCW N-2 reads + 2 writes
RAID6, N devices, N-2 of them data:
RMW 3 reads (old data, old P, old Q)
+ 3 writes (new data, new P, new Q) = 6 operations
RCW N-3 reads + 3 writes
The re-read is 4 KiB, not the chunk: md’s stripe unit is 4096 bytes, so the common claim that a small write re-reads a whole 512 KiB chunk is wrong. The multiplier is real all the same, roughly twice the host bytes on the drives for RAID5 and three times for RAID6.
The write hole, and the two fixes
md(4) is blunt: “interruption of write operations (system crash, etc.) to
RAID456 array can lead to inconsistent parity and data loss”. A write-intent
bitmap does not close it, and neither does drive-level power-loss protection,
because the hole is a failure across devices. Two mechanisms do. A journal
writes data and parity to a separate device first; in write-back mode it is a
real write cache that needs a PLP drive under it, and if the journal device dies
the array goes read-only. PPL (RAID5 only) stores partial parity in the
member metadata, and the
kernel documentation
puts its cost at write performance “reduced by up to 30%-40%”. PPL and a
write-intent bitmap cannot be used together, and PPL protects against silent
data corruption, not against losing in-flight data.
mdadm --create /dev/md0 --level=5 --raid-devices=6 --consistency-policy=ppl /dev/nvme[0-5]n1
mdadm --create /dev/md0 --level=5 --raid-devices=6 --write-journal /dev/nvme9n1 /dev/nvme[0-5]n1
echo write-back > /sys/block/md0/md/journal_mode
Raise the rebuild limit
Three settings to change on flash, starting values rather than answers:
echo 8192 > /sys/block/md0/md/stripe_cache_size
echo 4 > /sys/block/md0/md/group_thread_cnt
echo 2000000 > /proc/sys/dev/raid/speed_limit_max
The last line matters most. speed_limit_max defaults to 200000 KiB/s, roughly
200 MB/s, per device; on a 4 TB NVMe member it is the difference between a
rebuild of over five hours and one of about 22 minutes if the replacement
sustains 3.0 GB/s, as
rebuild time works through. The
other two spread the
single parity thread and enlarge its cache, at a RAM cost per member; both
depend on core count and workload, so measure rather than copy a number.
Discard, and the monitor
| Level | TRIM handling |
|---|---|
| RAID0 | Native, raid0_handle_discard() |
| RAID1 | Passed through; write-behind not used for discard |
| RAID10 | raid10_handle_discard(), rewritten in Linux 5.13 |
| RAID4/5/6 | Off by default |
Parity levels refuse discard by default because a region that does not read
back as zeroes poisons the next parity calculation. The drive-side field that
settles it is DLFEAT in nvme id-ns, where 001b means deallocated blocks read
as zeroes, and only 001b justifies setting
devices_handle_discard_safely.
md degrades quietly: it keeps serving reads until the second drive goes:
mdadm --monitor --scan --daemonise --test
echo check > /sys/block/md0/md/sync_action
cat /sys/block/md0/md/mismatch_cnt
mdadm /dev/md0 --replace /dev/nvme3n1 --with /dev/nvme9n1
--test sends a message at start-up, the only way to prove the alert path.
Set MAILADDR or PROGRAM in mdadm.conf first (its path varies by
distribution), or the monitor has nowhere to send and will not start.
check scrubs and counts into mismatch_cnt, and --replace --with swaps a
suspect member while it can still be read, so the array never goes degraded.
ZFS on NVMe: mirrors, ashift and recordsize
Two ZFS decisions are permanent for the life of a vdev - ashift and RAIDZ geometry - and the most consequential tunable is recordsize.
ashift and the layout
ashift is the base-2 logarithm of the smallest unit ZFS writes, fixed when the
vdev is created. Too low, and every small write becomes a read-modify-write
inside the SSD; too high, and every block is padded. NVMe namespaces often
report 512-byte blocks by default while also supporting a 4096-byte format,
which nvme id-ns lists. ashift=12 (4 KiB) is the defensible default.
Random IOPS scale with the number of top-level vdevs, not drives. Eight drives
in one RAIDZ2 vdev deliver roughly one vdev’s worth of random-read IOPS; the same
eight as four mirrors deliver four. Mirrors also rebuild better, since OpenZFS
can resilver a mirror sequentially (zpool replace -s) while a RAIDZ vdev walks
the block-pointer tree, and a mirror vdev can be removed later where a RAIDZ
vdev cannot, provided every top-level vdev shares the same ashift and any
encrypted datasets have their keys loaded. They cost 50 percent of raw capacity. The drive-buying half of the
decision is in drives for a NAS.
OpenZFS 2.3 added RAIDZ expansion with zpool attach, one disk at a time, but
existing blocks keep the data-to-parity ratio they were written with:
5-wide raidz1, 1 MiB of user data: 1 MiB x (5/4) = 1.25 MiB occupied
6-wide raidz1, same data rewritten: 1 MiB x (6/5) = 1.20 MiB occupied
the reflow alone recovers none of that 0.05 MiB
recordsize, and the padding nobody plans for
recordsize is a maximum, 128 KiB by default, and a write smaller than the record forces the whole record to be read and rewritten:
recordsize=1M, application writes 8 KiB at a random offset
read 1 MiB (unless the record is already in ARC)
write 1 MiB to newly allocated space
amplification against NAND: 1048576 / 8192 = 128x
recordsize=16K, same 8 KiB write
read 16 KiB, write 16 KiB, amplification 2x
Large records suit data written once and read in bulk; random overwrites want a
record close to the application’s page, which points at 8 KiB for PostgreSQL
and 16 KiB for InnoDB as a starting point. RAIDZ then rounds every allocation up
to a multiple of nparity + 1 sectors. At ashift=12, ten drives, RAIDZ2:
4 KiB block D=1 P=2*ceil(1/8)=2 3 -> roundup(3,3)= 3 sec = 12 KiB 3.00x
8 KiB block D=2 P=2*ceil(2/8)=2 4 -> roundup(4,3)= 6 sec = 24 KiB 3.00x
16 KiB block D=4 P=2*ceil(4/8)=2 6 -> roundup(6,3)= 6 sec = 24 KiB 1.50x
128 KiB block D=32 P=2*ceil(32/8)=8 40 -> roundup(40,3)=42 sec = 168 KiB 1.31x
The 1.25x that the width promises is approached only by large records. A zvol at
volblocksize=8K, the default until OpenZFS 2.2 raised it to 16K, returns 33
percent of raw capacity before compression, not 80. Usable capacity by layout
is in how much storage you need, and the
decimal-against-binary gap in why 4TB shows as 3.63TB.
The layers multiply into drive wear:
4 KiB application overwrite, recordsize=128K, 10-wide RAIDZ2, ashift=12
copy-on-write 4 KiB -> 128 KiB record rewritten 32x
RAIDZ allocation 168 KiB written to the vdev 1.31x
------------------------------------------------------------
allocated across the vdev 42 KiB per KiB of application 42x
... and the FTL's own amplification then applies on top
Endurance budget, ten 1.92 TB drives each rated 1 DWPD over 5 years:
1.92 TB * 365 * 5 = 3504 TB per drive, 35040 TB over the ten
35040 / 42 = 834 TB of application data
same sum at recordsize=16K (6x): 5840 TB
Parity arrays on flash are where enterprise endurance stops being optional.
Settings, caching and what ZFS cannot undo
compression=onselects lz4, which moves fewer bytes across PCIe and to NAND and gives up early on data that will not compress.autotrim=onis off by default, and without trim write amplification climbs as the pool fills.sync=disabledloses acknowledged writes back to the last transaction group, 5 seconds by default; leavesync=standard.- Free space: OpenZFS’s tuning guide asks for more than ten percent free; eighty percent full is the common rule of thumb.
A mirrored special vdev on two small NVMe drives moves a hard-drive pool’s metadata onto flash, but lose it and the pool is gone. SLOG and L2ARC are usually the wrong purchase on an all-flash pool, and NVMe caching is the home for all three. ARC, ZFS’s cache in RAM, defaults on current OpenZFS to the larger of all memory minus 1 GiB and 5/8 of memory, and the “1 GB of RAM per TB” line is dedup-era folklore; RAM caching for mechanical drives works through how much a pool can use. And plan the layout as if it were final, because much of it is:
| Operation | Status |
|---|---|
| Remove a mirror or single-disk top-level vdev | Possible since OpenZFS 0.8, and it leaves a permanent indirect mapping, held in memory, for the space that moved |
| Remove a RAIDZ top-level vdev | Not possible |
| Remove a top-level data vdev from a pool that contains a RAIDZ vdev | Not possible; cache, log and spare devices can still be removed |
| Shrink a vdev or a pool | Not possible |
| Widen a RAIDZ vdev by one disk | zpool attach, since OpenZFS 2.3 |
| Narrow a RAIDZ vdev | Not possible |
| Change parity level, RAIDZ1 to RAIDZ2 | Not possible |
Change ashift after vdev creation |
Not possible |
Every “not possible” row has the same workaround, zfs send to a new pool,
which needs the capacity twice over. For 4 KiB to 16 KiB random overwrites on
data you can rebuild, md RAID10 under XFS costs two writes per write and pads
nothing, against the 42x above. What you give up is end-to-end checksums,
snapshots and the detection of silent corruption.
btrfs, XFS, ext4 and the rest
A filesystem gives you redundancy only if it manages more than one device itself. btrfs and ZFS do. XFS, ext4 and f2fs do not, so their redundancy comes from md or LVM underneath.
| Stack | Data checksums | Own redundancy | Snapshots | Shrink | Write hole |
|---|---|---|---|---|---|
| ZFS mirror or raidz | yes | yes | yes | limited, no raidz | no |
| btrfs raid1 / 1c3 / 10 | yes | yes | yes | yes | no |
| btrfs raid5 / raid6 | yes | yes | yes | yes | yes, documented |
| bcachefs with replicas | yes | yes | yes | yes | no |
| XFS on md RAID10 | metadata only | no | no | last AG only | none, no parity |
| XFS on md RAID5/6 | metadata only | no | no | last AG only | yes, unless a write journal, or PPL on RAID5 |
| ext4 on md RAID1 | metadata only | no | no | yes, offline | none, no parity |
| LVM RAID + integrity | yes | yes | yes | not while integrity is on | parity levels inherit md’s |
| f2fs, single device | metadata only | no | no | no | not applicable |
Decide the data checksum column first. ZFS, btrfs and bcachefs make it a
mkfs decision you cannot revisit, while dm-integrity can be added to an existing
LVM RAID volume with lvconvert --raidintegrity y.
- btrfs: the mirrored profiles are good, take mismatched drive sizes, and
repair a bad copy from the good one. btrfs RAID5 and RAID6 carry a write
hole, and the project documents it: the
upstream status page
marks RAID56 unstable. Use
discard=async. - XFS on md is the boring high-performance answer, and mkfs.xfs reads md’s geometry by itself. Data is not checksummed.
- ext4 remains right for a boot volume on a two-drive md RAID1.
- bcachefs has the right design, but it was removed from the kernel in 6.18 and ships as an out-of-tree DKMS module, a poor bet for storage you must keep for five years.
Endurance, power-loss protection and heat
The drive that fails you in an array is rarely the one that dies. It is the one that was never rated for the workload an array produces. SSD endurance is the home for TBW and DWPD; this is what an array adds.
Parity spends the rating
Application writes 1.00
Filesystem metadata and journal x1.1
RAID6 partial-stripe parity x3
--------------------------------------------------------
Host bytes written per application byte ~3.3
Drive: 1.92 TB, rated 1 DWPD over 5 years
rated host writes = 1 x 1.92 TB x 365 x 5 = 3,504 TB = 3.5 PB
application budget = 3,504 / 3.3 = 1,062 TB
effective rating = 1,062 / 3,504 = 0.30 DWPD
A DWPD figure already contains the drive’s own amplification under the JEDEC
qualification workload, but not yours: an array kept 90 percent full and written
randomly can amplify harder, in which case 0.30 DWPD is the best case, while a
large sequential workload can run the other way. Spare area is free
if you take it before partitioning: run blkdiscard over the namespace and
partition short of the end.
The SLC cache and power-loss protection
An advertised sequential figure is a burst figure. An array meets the steady-state figure. Consumer TLC and QLC drives write first into a cache that shrinks as the drive fills, and a rebuild or bulk ingest is a sustained write to every member at once, so the whole array leaves its cache in the same minute.
PLP guarantees only that data the drive has acknowledged stays acknowledged, and it does not close the RAID5/6 write hole. Its everyday value is speed under sync writes, because a PLP drive completes a flush from a power-safe buffer:
Single-threaded synchronous writer, latency bound.
The flush latencies below are assumed, not measured:
sync writes per second = 1 / flush latency
PLP drive, flush satisfied from a power-safe buffer
~20 us -> ~50,000 sync writes/s
no-PLP drive, flush waits on a TLC program
~800 us -> ~1,250 sync writes/s
Those are order-of-magnitude figures, not datasheet values, but every fsync and database commit lands on the second line. Shop for PLP with a filter, enterprise-class SSDs, then look for it on the datasheet; its absence is the answer.
Identical drives wear out identically
Mirror members take identical writes and share a firmware image. HPE’s customer bulletin of November 2019 is the cleanest example: a supplier firmware defect made certain SAS SSDs fail at 32,768 power-on hours, which HPE put at 3 years, 270 days and 8 hours, unless firmware HPD8 had been applied, and a March 2020 advisory covered drives failing at 40,000 hours. Same model, same batch, same firmware is one failure domain wearing several badges. Split the order across two sellers or batches, or run two models of the same capacity; enterprise-class drives of any kind ranked by price per terabyte make the second model easy to pick.
Heat and power
Mass buys the benchmark, air buys the array. A heatsink’s mass decides how
long a drive runs before throttling; airflow decides where it settles. A hot
controller drops to a lower power state and stays there for as long as the heat
has nowhere to go, and nvme smart-log records the minutes spent above its
temperature thresholds.
M.2 has no 12 V rail, and a PCIe slot supplies 3.0 A on 3.3 V, which is 9.9 W:
Four M.2 drives, Samsung 990 PRO 2 TB datasheet figures (Rev 2.0)
average active read 4 x 6.1 W = 24.4 W
that current at 3.3 V = 7.4 A
PCIe slot 3.3 V budget = 3.0 A
Those are Samsung’s average active figures for the 990 PRO 2 TB from its datasheet, Revision 2.0 of July 2023. A carrier that takes 3.3 V straight from the slot, with no 12 V regulator of its own, can feed one drive at that rate, and not four.
Eight drives, all active, continuous
consumer M.2 at 5.5 W write each = 44.0 W
enterprise U.2 at 15 W read each = 120.0 W
Solidigm’s brief quotes 15 W average active read for the 3.84 TB D7-P5520. A hundred and twenty watts of continuous drive load never spins down. Idle is a platform question: one documented Z590 build measured the whole system, with no cards fitted, at 28 W on BIOS defaults and 16 W once PCIe ASPM worked and runtime tuning got the CPU package to C8, and refitting a ConnectX-3 NIC or an LSI HBA, neither of which will take ASPM, put it back to 25 W to 26 W.
Failure and rebuild on flash
The argument that killed RAID 5 was an argument about hard drives. It rested on the consumer SATA bound of no more than one unrecoverable read error per 10^14 bits read:
Consumer SATA UBER bound 1 error per 1e14 bits read
1e14 bits / 8 = 1.25e13 bytes = 12.5 TB
Rebuild of a 6 x 2 TB RAID 5 reads the 5 survivors in full
5 x 2 TB = 1.0e13 bytes = 8.0e13 bits
P(at least one error) = 1 - (1 - 1e-14) ^ 8.0e13 = 0.55
Flash changed the inputs. The UBER figure is a specification bound, not a measured rate. Data-center NVMe datasheets quote one unrecoverable sector per 10^17 bits read, Samsung’s PM1725b among them, which puts the same rebuild at about one chance in 1,250. And Linux md keeps a 4 KiB bad block list per member, so an unreadable block during recovery costs a block, not the array. The risk moved into correlation: members wear in lockstep, share firmware, and a worn SSD that stops accepting writes while still serving reads is ejected by md all the same.
Rebuild time, and the default worth raising
Member 4.0 TB = 4.0e12 bytes
md stock speed_limit_max 200,000 KiB/s = 204.8 MB/s
4.0e12 / 204.8e6 = 19,531 s = 5 h 25 min
Cap lifted, bounded instead by the replacement's sustained
write rate past its SLC cache, say 3.0 GB/s
4.0e12 / 3.0e9 = 1,333 s = 22 min
The 3.0 GB/s is a stand-in: substitute the sustained write figure of the drive
you are fitting, because past the SLC cache it varies by an order of magnitude
across the NVMe drives listed here. A write-intent bitmap, which
recent mdadm adds only when asked with --bitmap=internal, lets a member that
dropped out briefly be re-added with only the dirty regions recovered; it
cannot be combined with PPL.
When a member fails
When md ejects a member, read the drive before you replace it:
nvme smart-log /dev/nvme0 # critical_warning, percentage_used,
# available_spare, media_errors
nvme error-log /dev/nvme0 # the controller's own recent errors
nvme id-ns /dev/nvme0n1 # nsattr bit 0: namespace write protected
smartctl -a /dev/nvme0 # the same log, different presentation
lsblk -o NAME,RO,SIZE # RO=1 is the read-only transition, visible
cat /proc/mdstat # which member md ejected
In critical_warning, bit 3 means media placed in read-only mode, and it is the
bit that ejects an otherwise healthy member. available_spare moves first;
what SMART tells you and how to read it covers
the rest of the log. Scrub monthly, which costs flash no endurance: Debian and
Ubuntu ship checkarray, and mdadm ships mdcheck_start.timer, so check which
one your distribution actually enabled before assuming either runs. A nonzero
mismatch_cnt on RAID 1 or RAID 10 is not by itself corruption; on RAID 5 or 6
it wants investigating.
A redundant array is not a backup. Every write reaches every copy at the same
instant, including a mistaken rm -rf, an encrypting process and a filesystem
bug. Redundancy covers a device stopping. It covers nothing else.
Buying the drives
Buy the counters, not the drive. An NVMe SSD arrives carrying a written
record of how hard its last owner used it. Ask the seller for the output of
nvme smart-log or a CrystalDiskInfo screenshot before paying; a seller with the
drive in a machine produces it in a minute, and the report is the reading: buy a
drive without one only at a price that allows for it.
Buying used drives on eBay covers the checks
in full, including how Data Units Written converts to terabytes.
The traps, one line each
| Trap | What actually happens |
|---|---|
| B+M keyed M.2 drive | Links at PCIe x2 or SATA, never x4 |
| M.2 SATA sold as “M.2 SSD” | Same board outline as NVMe, and a B+M notch fits either, so only the model number settles the protocol |
| M.2 22110 in a 2280 board | 30 mm too long; enterprise M.2 with power-loss protection is often 22110 |
| U.2 drive, U.3 backplane | SFF-TA-1001 says no link; the reverse is mandated to work, and some backplanes are wired both ways, so check the vendor document |
| E1.S or E3.S | SFF-TA-1002 connector; no EDSFF bay in a desktop, see EDSFF in the interfaces guide |
| Passive quad-M.2 card | Without x4x4x4x4 bifurcation in firmware, one drive enumerates |
| SAS SSD in a 2.5-inch photograph | Not NVMe, and no NVMe host will see it |
What a pulled enterprise drive needs
Pulls are the value play: a 3 DWPD drive retired at 20 percent of its life can have more writing left in it than a new consumer drive has in total. Read that rating off the model’s datasheet, because the capacity does not give it away. Micron’s 7450 product brief (Rev. C, February 2026) rates the 7450 MAX “Mixed-Use, 3 Drive Writes per Day”, its 400 GB and 800 GB M.2 models included, and Kioxia’s CD8-V starts at 800 GB at 3 DWPD, while Kioxia’s SAS PM6-M is 10 DWPD from 400 GB to 3.2 TB (both Kioxia briefs Rev. 1.2, 2023). SSD endurance is the home for DWPD.
The host owes the drive 12 V and up to about 25 W, forced air, and the right
cable, which for a Gen3 or Gen4 pull is the easy case. A pull also arrives in
whatever sector format the last array wanted.
nvme format /dev/nvme0n1 --lbaf=N changes it, destroying the namespace
contents, but OEM-badged firmware is widely reported to refuse the operation or
to offer only metadata-bearing formats, and a drive that refuses stays in the
format it arrived in. If the listing shows a Dell, HPE or NetApp part number, ask
the seller to confirm that nvme id-ns lists a format with ms:0 (no per-sector
metadata; a metadata format presents 520- or 4104-byte sectors many hosts will
not use) before money moves.
A large drive priced far under its size page usually has one of three explanations: it is counterfeit, which a full write and read-back verify catches; it is QLC sold on its burst figure, slowest at the one moment the array depends on it; or it is at eighty percent of rated life, which the drive’s own counters show, so ask for them.
Rank by what the drive has left to write
Every NVMe SSD on the site is ranked by price per terabyte, and every drive puts NVMe next to SAS and hard drives on one scale. For an array member, divide by the writing the drive has left. Two illustrative parts, at round prices rather than quotes:
A: new 2 TB consumer TLC, 1,200 TBW rated, $90
90 / 1200 = $0.075 per TB written
B: 1.92 TB enterprise pull, 3 DWPD over 5 y = 10,512 TBW rated,
Percentage Used 18 -> 8,620 TBW left, $140
140 / 8620 = $0.016 per TB written
The pull costs about half as much again on the shelf, and its cost per terabyte written is a little over a fifth of the new drive’s. Build the shortlist by price per terabyte, re-sort it by the second figure, and check how prices are moving on Trends before buying.