Hello and welcome back!
Today we’re tackling one of the most important — and honestly, most exciting — parts of building a serious storage system: getting your very first OpenZFS pool architecture right.
We’re going to demystify vdevs, mirrors, RAID-Z, and the permanent decisions you’ll make in the first thirty minutes that will affect your storage for the next five to ten years.
Because here’s the thing I’ve learned after watching far too many people (including my younger self) get this wrong: you can’t easily fix a bad foundation. You can add more capacity later, but you cannot reshape the concrete once it’s poured. So let’s pour it correctly — together.
The Big Picture: Pools vs vdevs
Let me paint a vivid picture.
Think of your storage pool as a massive water reservoir that holds all your data. The individual vdevs (virtual devices) are the actual tanks that make up that reservoir.
Here’s the crucial part most people miss:
- You can always add more tanks to make the reservoir bigger.
- You cannot easily change the size, type, or layout of a tank once you’ve put it in the ground.
This is why your vdev design is quite literally a permanent, foundational decision. We’re not just choosing disks — we’re choosing the DNA of your entire storage system.
The Two Main vdev Flavors
ZFS gives us two primary ways to build a vdev. Each has its own personality, strengths, and trade-offs.
The Mirror: Speed and Simplicity
A mirror vdev is exactly what it sounds like — an exact copy of your data across two or more disks. Every write goes to all of them simultaneously.
I’m genuinely fond of mirrors. There’s something beautiful about their elegance. Random write performance is excellent (perfect for VMs, databases, or anything with lots of small writes), and if a drive fails, recovery is almost boringly simple. The system just keeps reading from the surviving twin while you swap the dead drive.
The downside is obvious: you only get the capacity of a single drive. Two 18TB drives in a mirror only give you 18TB of usable space. It feels expensive — until your VM cluster is flying and your rebuilds take minutes instead of days.
My rule of thumb: Use mirrors when performance and fast recovery matter more than capacity.
RAID-Z: The Space-Efficient Beast
When you need lots of storage, RAID-Z is where the magic happens.
Instead of making full copies, RAID-Z uses clever parity calculations (the same idea as traditional RAID but done with ZFS’s characteristic intelligence). This lets you survive drive failures while maintaining much better capacity efficiency.
You have three main options:
– RAID-Z1: Survives 1 drive failure
– RAID-Z2: Survives 2 drive failures (my usual minimum for anything serious)
– RAID-Z3: Survives 3 drive failures (for when you’re feeling very paranoid)
The capacity savings can be dramatic. Eight 18TB drives in RAID-Z2 give you roughly 108TB of usable space instead of just 36TB in mirrors. That’s real money.
However, as my grandmother used to say, there’s no such thing as a free lunch.
RAID-Z requires more computational work for every write. More importantly, when a drive fails, the resilver (rebuild) process can take days on large drives. During that time, your remaining drives are working overtime, dramatically increasing the chance of a second failure. This is the stuff that keeps storage nerds up at night.
How Many Disks Should You Put in a vdev?
This is the question I get asked most often.
For mirrors: Keep it simple. Two or three disks per vdev is perfect. I’m particularly fond of two-disk mirrors.
For RAID-Z, here are my practical guidelines:
– RAID-Z1: 3–6 disks (and honestly, I rarely recommend RAID-Z1 anymore)
– RAID-Z2: 6–10 disks is the sweet spot
– RAID-Z3: Usually 8–12 disks
Here’s my unbreakable rule: avoid “wide” vdevs like the plague. A 20-disk RAID-Z2 vdev might look efficient on a spreadsheet. In real life, it’s a high-stakes gamble every time you resilver.
The Art of Combining vdevs
Now for the fun part — how do we actually build the pool?
Your pool stripes data across all top-level vdevs. This is where performance comes from. And here’s my strongest recommendation:
Build your pool from multiple identical vdevs.
Instead of one massive 12-disk RAID-Z2 vdev, create two separate 6-disk RAID-Z2 vdevs. You’ll get better performance, dramatically lower risk, and a beautiful expansion path. When you need more space, you simply add another identical 6-disk RAID-Z2 vdev.
This modular approach is pure joy to work with.
Common Pitfalls (Learn From My Mistakes)
Let me save you some pain by sharing the greatest hits of expensive lessons I’ve watched over the years:
- The “One Giant vdev” Trap: Putting every single drive into one massive RAID-Z vdev. It feels clever until you need to expand or you’re staring at a 4-day resilver with your heart in your throat.
- Using RAID-Z1 with modern drives: Anything over 4TB in RAID-Z1 is playing with fire. The resilver times are just too long. I now consider RAID-Z2 the absolute minimum for bulk storage.
- Forgetting permanence: You cannot add a single disk to an existing RAID-Z vdev. Ever. The layout is carved in digital stone. Plan accordingly.
Real-World Architectures That Actually Work
Home Media Server: Capacity is king. A single 6- or 8-disk RAID-Z2 vdev is often perfect. Simple, cost-effective, and plenty fast for streaming.
Small Business VM Host: Performance matters most. I love building these from multiple two-disk mirror vdevs. The responsiveness is fantastic and rebuilds are quick.
Creative Workstations (Video Editors, etc.): The Goldilocks zone. Two smaller RAID-Z2 vdevs often gives you the perfect blend of capacity, performance, and future expandability.
The Most Important Takeaway
Your storage pool is only as good as the vdevs it’s built from. Mirrors are fast but hungry for space. RAID-Z is efficient but demands respect for its resilver characteristics.
Plan your vdevs like you’re designing something permanent — because you are.
Get this part right and everything that comes after (snapshots, replication, encryption, tuning) becomes so much more enjoyable.
And that, my friends, brings us to the perfect cliffhanger.
Now that we have a solid, resilient foundation, how do we protect ourselves from the most dangerous threat in storage — ourselves? (And ransomware, of course.)
Next week in Episode 6, we’re diving into one of ZFS’s most beloved features: instantaneous snapshots and how they can give you a genuine “Get Out of Jail Free” card.
You won’t want to miss it.
Until then, take your time with your vdev planning. Your future self will thank you.
What’s the most interesting storage pool you’ve ever built? Drop your architectures (and war stories) in the comments — I read every single one.
Talk soon,
Your fellow storage enthusiast