Home Rack Colo Project - ETA To Complete June '23

TheTamago said:

4x Seagate EXOS or IW 14TB(16TB?) in RAID10(ZFS) via TrueNAS

Click to expand...

Anything other than RAID-Z1 seems wasteful to me. Presumably, you have a backup, and that covers the R part of RAID. I keep a single parity drive simply for convenience in case one drive goes down. Its actually faster to restore a backup using zfs send-recv on a 10GbE network than it is to order a new drive, have it arrive in the mail, and then rebuild the array. Especially once you are talking about modern high capacity drives.

I have a similar setup, 5 14TB Exos drives, a single parity. RAID10 means you are losing half the capacity for no real reason, IMHO. Im even virtualizing TrueNAS in ESXi, lol. Using a Dell R710 though because RAM and not CPU is the most important part of my workloads.

FYI, Ive been tracking these for replacements and it looks like 16TB is now cheaper than 14tb.

Given the risk of an uncorrectable bit error with such large volume sizes we normally prefer 2-drive parity ... but yeah... with just four disks that is a 50% capacity hit. Depends on what the goal is here, if it's uptime, then I would still do it. But if the goal is merely a larger-than-a-single-disk-can-do volume and uptime is not a key goal.....

At these huge drive capacities, RAID rebuild (resilver) times are considerable even with the optimizations that have been done to speed up resilver times. I've replaced a 18TB drive in my 6 x RaidZ2 array, and it took almost 24 hours. If another drive even so much as burps during a resilver, you stand the chance of losing the while pool if you only have RAIDZ1, and a resilver is very hard on the drives as they are running continuously during the rebuild and if you access the array during a resilver, there is a LOT of thrashing. The cost of that extra drive is negligible, IMHO, compared with the other hardware costs and hassle of restoring data. It's $300 for an 18TB drive that will function for, say, 5 years. That's $5/month.

Conventional wisdom states that RAIDZ10 is supposedly faster than RAIDZ1 or Z2 (since it's pure striping with no parity), but I don't really see a practical impact. Any VMs should be hosted on SSDs anyway (IMHO), and the spinners should be used for bulk storage. RAIDZ2 will allow any 2 drives to fail, while RAIDZ10 might get hosed depending on which 2 drives fail. RAIDZ2 has the potential of being expanded later with more slices, while that's not possible with RAIDZ10. I'd go with RAIDZ2.

You might regret the lack of PCI slots in a micro-ATX board. Your drive count obviously maxes out your SATA headers, so you might find yourself needing to buy an HBA card. Add in another card later for, say, a 10Gb NIC and you're out of space.

You are also not taking advantage of the M.2 slot on the board. I'd use that for the ESXI Host, leave the SSDs for VMs and spinning rust for bulk/ local backups.

My server sucks down ~130W, and it has had a huge impact on my power bill. I don't have AC (don't really need it in my area), so this doesn't take into account secondary cooling costs of running an AC to manage heat. The extra power draw pushes my electrical usage into a "high usage" tier, so the the real per-KWh rate that it takes to run the server is fairly steep. Obviously, YMMV and your electric costs are probably different, etc. but don't forget to take this into account. I did the math (napkin-level, but close enough), and the increase in my electric bill actually adds up to be more than my on-going hardware purchase/replacement costs.

I run ESXI on bare metal on an SSD. ESXI runs some VMS on other SSDs, and one of my VMs is TrueNAS. I have 2 x pools in TrueNAS running on 2 x LSI controllers that I passed through to ESXI, so TrueNAS is managing the pools directly. The TrueNAS VM is the first VM to boot, so it's ready by the time the other VMs boot up and need to see the mapped drives that they then use for bulk storage (mostly multimedia and files). My ESXi config is backed up and I image save an image of the boot drive to a USB drive before every ESXi patch or update (which I do about every 6 months). I found this arrangement to be simpler and more cost efficient than trying to optimize the ZFS pool to host VMs on directly - I screwed around with ZIL and SLOG drives and it really wasn't worth it. Not saying that this is the "right" way, but throwing it out there for consideration. I've had this setup running for almost 10 years now with no data loss and survived several drive failures, dozens of ESXi updates/upgrades (ESXi 5.0 --> ESXi 7), expanded my drive capacity almost ten-fold, and it's been 100% rock solid.

The only thing I can think of is if there's a need for that much storage space. I mean, 4x 16TB in a Raid 10 is still something like 30 TB or so. that's a LOT of storage.

IMHO, it's a lab setup at home - is there anything that specfically is so huge it needs that much space? And with a RAID10, no hot spare?

If you don't have an active plan for that much storage (like, I guess replicating very large databases at home or something), I'd suggest exploring using SSDs in a RAID6 or something. Or split your storage and have a "fast" and "slow" tiers. For me personally, I'd rather have a hot spare available and I'd rather go with a higher bandwidth and lower latency on storage. It's like night and day when hosting VMs.

As an anecdote - when we migrated our work ESX hosts from using a NetApp with 3.5" spinning disks to a Pure SAN array using Intel SSDs, the latency went from 200+ milliseconds to less than 2 milliseconds.

caustic meatloaf said:

As an anecdote - when we migrated our work ESX hosts from using a NetApp with 3.5" spinning disks to a Pure SAN array using Intel SSDs, the latency went from 200+ milliseconds to less than 2 milliseconds.

Click to expand...

If you didn't put an extra 0 in there, and that isn't highest latency seen for a single IO (a somewhat worthless metric), but actually a meaningful measure, the issue wasn't exactly spinning disk. The issue was way under-speccing your equipment. Well, even 20ms 95th percentile is underspecced, but in the realm of "I've seen it before, but it *really* needed fixed".

You might want to look into VMUG advantage. That is 200 a year and gets you access to pretty much all of VMware's stuff. Also you can usually find a 10 or 20% off coupon.

What's the price gap between the Zen 2 and Zen 3 stuff? (especially Zen 4 is now out)

7302P is 16 cores so not sure how that compares to the desktop stuff, but if you want all the PCI-e lanes then EPYC definitely is the way to go.

continuum said:

What's the price gap between the Zen 2 and Zen 3 stuff? (especially Zen 4 is now out)

7302P is 16 cores so not sure how that compares to the desktop stuff, but if you want all the PCI-e lanes then EPYC definitely is the way to go.

Click to expand...

Who can pass up a motherboard having 5 x PCIe4 X16 slots!? I don;t need this and I'm not sure I can afford this, but DAMN is this a good base for a home server build. My only concern would be power use and efficiency and heat.

My biggest concern with this build is the fact that it's based on an AMD Ryzen platform with is NOT one of the platforms officially supported by VMWare. There are issues with passthough that can be worked around, but I'd be paranoid about some future update/patch to ESXi potentially breraking that workaround or introducing other issues.

I have a really old ESXi box running 4-5 VMs including a TrueNAS VM with a couple SAS cards in passthrough (Haswell-era Xeon 1245V3 on a Supermicro X10SL7 board). It's getting really dated, the 32Gb RAM limit is becoming an issue, and the CPU is no longer supported by VMWare, but I did get about 10 years out of the system that was 100% trouble-free and did not require any work-arounds for things becuase it is a well-supported platform. When I plan my upgrade, I'll likely stay with Intel just due to that fact. As much as I would have loved to jump to AMD, I don't want that Damocle's Sword of suddenly requiring hours of time on on a weekend to figure out how to make an ESXi update work becuase yet another workaround/bandaid broke at some point.

With storage having a lot a of capacity is all good but I would say faster rather than bigger is always the way to go with virtualisation.
In my current lab I've got a few templates in various OSes, for instance, Windows Server 2022 template. A few powershell scripts that will render out a quick AD domain with a CA, and a few scripts to populate the AD with accounts etc for testing. Others with Debian, Rocky, Ubuntu etc.
With automation speed is a the real win.
Spinning rust, even commercial is not as fast as pcie nvme consumer.
Fast network using bonded links, lots of memory, and more cheaper nodes rather than 1 monolithic point of failure.
Currently running esxi 8 on a pair of HP ProDesk 600 G3 SFF, with 64GB of DDR4 RAM and i5 processors in each.
They are managed by vcsa 8.
Licenses are VMUG ones couple of hundred bucks a year. With that you get all the stuff you'll want to play with licenses that last the length of of membership (1 year)
In the process of putting in some SFP+ broadcom 10GBe's into them, they've got a couple of ports so one for storage and one for vMotion
Best toy I recently got was a Asustor Flashstor 12. All flash NAS. Its not production but it lifts for a homelab
Its running TrueNAS Scale.
It's running raidz1 and its got about 10TB available 7 wide, and its got dedup on as well. Virtualisation is one of the best use cases for it. As its all flash, I've had no issues so far and I haven't even leveraged the 10GBe bits yet. It will clone out a 2022 server template in comfortably under a minute, I expect that might improve once its all 10GBe.
10Tb is a lot of VMs if you go by minimum spec for the bulk of system roles if you take out of the equation actual live data.
If you are using this as a test bed, there won't be any real data to speak of.
The minimum spec for a lot stuff is tiny for storage.