RAID-Z, RAID-Z2 and RAID-Z3: how ZFS parity levels differ
RAID-Z is ZFS's integrated parity RAID. Unlike a hardware controller sitting beneath a file system, the file system and the RAID layer are the same thing: ZFS writes variable-width stripes using copy-on-write, so a stripe is either fully committed or not committed at all. That removes the classic parity write hole, and it also changes what a failed pool looks like in the laboratory. The tables below compare the three parity levels, contrast RAID-Z with conventional RAID 5 and 6, and list the failure modes we actually see on ZFS pools.
RAID-Z parity levels compared
| Level | Parity devices per vdev | Minimum disks (practical) | Disks that can fail | Usable capacity (n disks) | Resilver exposure | Typical use |
|---|---|---|---|---|---|---|
| RAID-Z1 (RAID-Z) | 1 | 3 | 1 | (n − 1)/n | A second disk failure during resilver loses the vdev, and with it the whole pool | Small pools built from small, low-latency disks |
| RAID-Z2 | 2 | 4 | 2 | (n − 2)/n | Still tolerates one further failure while resilvering | The general-purpose choice for most production pools |
| RAID-Z3 | 3 | 5 | 3 | (n − 3)/n | Tolerates two further failures while resilvering | Wide vdevs and high-capacity disks where resilvers run for days |
RAID-Z compared with conventional RAID 5 and RAID 6
| Property | RAID-Z (ZFS) | Hardware / software RAID 5 and 6 |
|---|---|---|
| Parity write hole | Not present — copy-on-write means a stripe is never partially updated in place | Present unless the controller has a battery- or flash-backed cache |
| Stripe width | Variable; each record is written across as many disks as it needs | Fixed stripe size configured at array creation |
| Integrity checking | Every block is checksummed; scrub verifies and repairs from parity | Parity is normally only consulted during a rebuild; silent corruption can go undetected |
| Rebuild scope | Resilver copies only allocated blocks, so an empty pool resilvers quickly | Rebuild reads and writes every sector regardless of use |
| Layout knowledge needed for recovery | Read from the on-disk ZFS labels and uberblocks rather than vendor metadata | Member order, stripe size, parity rotation and offset must be solved |
| Expansion | Historically by adding whole vdevs; single-disk expansion depends on the ZFS implementation and version in use | Many controllers support online capacity expansion of an existing array |
| Effect of a controller change | Pools are portable between hosts because layout metadata lives on the disks | Often controller-specific; a different controller may not import the array |
RAID-Z failure modes and what determines recoverability
| What happened | Why it is difficult | What determines the outcome |
|---|---|---|
| More disks failed than the vdev has parity (2 on Z1, 3 on Z2, 4 on Z3) | The pool cannot be imported; a single unrecoverable vdev takes the whole pool offline | Whether the failed members can be individually imaged well enough to reconstruct the missing stripes |
| Resilver started and then failed or was interrupted | Partly rewritten state on the replacement device alongside a degraded vdev | Stop further resilver attempts; recovery works from images of the pre-resilver members |
| Pool labels or uberblocks damaged | ZFS refuses to import a pool whose metadata does not verify | Whether an earlier uberblock or a redundant label copy can be used as a rollback point |
| Accidental zpool destroy, dataset or zvol delete | Space is released and copy-on-write may reuse it on the next write | How much has been written since; the pool should be taken offline immediately |
| Snapshot or dataset removed and the pool kept in service | Freed blocks become allocation candidates | Time and write volume between the deletion and taking the pool offline |
| HBA, cable or expander corruption | Bad data is written and checksummed as valid on arrival | Scrub history and whether known-good copies of the affected blocks exist |
| Whole batch of identical disks aging out together | Multiple members reach their failure point within the same window, often during a resilver | How many members are still readable and how much of each can be imaged |
| ZFS native encryption or a self-encrypting disk set, key unavailable | The blocks can be recovered as ciphertext but not decoded | Availability of the encryption key or passphrase; without it the plaintext cannot be produced |
Notes
- Parity in RAID-Z is counted in devices, not percentages: RAID-Z2 survives any two device failures within a vdev regardless of how wide the vdev is.
- A pool is only as available as its least redundant vdev. Losing one vdev takes the entire pool offline, which is why mixing a RAID-Z1 vdev into an otherwise RAID-Z2 pool undoes the extra protection.
- Real usable capacity is lower than the (n − p)/n formula suggests. Sector size (ashift), record size and parity padding all consume space, and wide vdevs of small records lose proportionally more.
- RAID-Z3 exists because resilver windows grew with disk capacity. The longer a vdev runs degraded, the more likely a second or third failure lands inside that window.
- A degraded or unimportable pool should be imaged before any further import, resilver or repair attempt. Every additional attempt writes to the members and can remove rollback options.
- Snapshots and replication protect against deletion and corruption. Parity does not — RAID-Z faithfully preserves whatever the file system was told to write.