260 lines
14 KiB
Markdown
260 lines
14 KiB
Markdown
---
|
|
title: "Our Home Server Build-Out Log Part 1: Speccing It Out"
|
|
date: 2022-10-31T17:00:00+11:00
|
|
lastmod: 2022-11-06T10:45:00+11:00
|
|
draft: false
|
|
showSummary: true
|
|
summary: "We've wanted to build a home lab for a long time. Here's how it went."
|
|
series:
|
|
- "Home Server Build-Out"
|
|
series_order: 1
|
|
---
|
|
|
|
# Part One: Specification
|
|
|
|
## Background
|
|
|
|
We've wanted to build a home lab for a long, long time, but never got around to it.
|
|
|
|
> **(Selene)** However, a certain someone was complaining about disaster recovery and migrating workloads to us,
|
|
> and that turned out to be just the push we needed to get this rolling.
|
|
|
|
Since we also want to do more blogging and community contribution, why not make this the inaugural post on our new blog?
|
|
|
|
> **(Lorelai)** So, let's get this show on the road, shall we?
|
|
|
|
## The Use-Case
|
|
|
|
So what did we need from this server? After talking over with friends, the following use-cases came to mind:
|
|
- Storage Cluster (ZFS of some kind)
|
|
- Foundry VTT Server
|
|
- [chibisafe](https://github.com/chibisafe/chibisafe) Server
|
|
- Modded Minecraft Server ?
|
|
- Generic cloud host for friends
|
|
|
|
## The Spec
|
|
We wanted this server to be the basis for a home lab that could later be expanded, so we want it to have a decent amount of oomph.
|
|
After deliberating on it for a while, we decided on the following:
|
|
|
|
| Part | Spec |
|
|
| ----------- | ----------------------------------------------------------- |
|
|
| CPU | Min. 16 core/32 thread |
|
|
| Motherboard | Must support large amounts of RAM and have BMC/IPMI |
|
|
| RAM | Moderate speed, 128GB to start, more to be added |
|
|
| Storage | At least 32 TB usable, 2 redundant disks min., 1 hot spare |
|
|
| Networking | Min. 2 ✕ 1Gbps management, 2 ✕ 40Gbps data |
|
|
|
|
### But Why?
|
|
|
|
> **(Selene)** I'm no techie, but this all seems rather excessive for a home server. Especially compared to people we know.
|
|
>
|
|
> **(Ashe)** You're not wrong there, Selene. I could wax lyrical about how I want this server to last a long time and be
|
|
> useful for many different things and so on and so forth, but if we're being honest, the main reason is "Tech shiny!"
|
|
>
|
|
> **(Tammy)** So the same reason we've always gotten high spec PCs, really.
|
|
|
|
|
|
## Fulfilling the Spec
|
|
Now that we have a spec, we need to figure out the best way to fulfill it.
|
|
|
|
### The Case
|
|
This would normally be something to consider carefully, but a good friend was getting rid of a 3U server case with
|
|
12 3.5" bays, so we'll nick that off of him, and move on. He also donated us a few
|
|
[Arctic P12s](https://www.arctic.de/en/P12/ACFAN00118A) for cooling as they'd be quieter than the jet engines
|
|
that came with the case.
|
|
|
|
### The PSU
|
|
We have a spare 750W Silverstone 80+ Platinum PSU lying around the house from an old PC, so we'll just repurpose that here.
|
|
|
|
### The Storage
|
|
It's probably strange to start with the storage, but we'll need to know how much storage we have to spec out
|
|
the ZFS instance properly.
|
|
|
|
#### The Goal
|
|
The goal is to have a scaleable template that can be expanded without breaking the bank each time,
|
|
while maintaining favourable failure characteristics. We'll budget out 3000 AUD for each expansion.
|
|
|
|
> **(Doll)** That sounds like a lot, Miss...
|
|
>
|
|
> **(Ashe)** It's a good chunk, yes. But this should, ideally, only happen every few years, so we can amortise this
|
|
> over the life of the storage.
|
|
|
|
We have 12 3.5" slots in our 3U compute case, and most storage arrays are either 24 or 36 bay, so planning around
|
|
12 unit slices seems logical.
|
|
|
|
If we round up and call it 3600 AUD per slice, we get ~300 AUD per HDD.
|
|
At time of writing, the best $/GB is 16TB Seagate Exos drives. We picked up 6 a year ago when exchange rates
|
|
weren't so miserable, and will be picking up another 6 to round out this cluster after the new year.
|
|
This overshoots our budget by about 25%, but there's little to be done about exchange rates.
|
|
|
|
We can, however, expect these to fall in price over time,
|
|
so this will likely be within budget by the time we need another slice.
|
|
|
|
|
|
#### RAID Layout
|
|
[dRAID](https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html) came to OpenZFS a while back,
|
|
and has improved resilver times compared to Z1/Z2/Z3, as well as allowing for much more granular redundancy design,
|
|
so we'll be using that.
|
|
|
|
The Seagate Exos drives are specc'd at ~200MB/s peak write rate, which gives us a resilver time of just over 22 hours in
|
|
traditional Z1/Z2/Z3. And that's 22 hours of every other drive in the array getting hammered. This is the main reason
|
|
we're going to be using dRAID. Under dRAID, we specify a number of data, parity, and spare slices, which are distributed
|
|
amongst the physical drives.
|
|
|
|
In our case, we want at least one, maybe two, hot-spare slices to make best use of dRAID.
|
|
This leaves us with 10 or 11 slices to play with.
|
|
|
|
With 11 usable slices our best bet would probably be 8/3/1 (8 data, 3 parity, 1 hot-spare), where as with 10 usable slices,
|
|
we'd probably want to go for either 7/3/2, or 8/2/2.
|
|
|
|
7/3/2 is certainly the safest set-up, as it gives us 3 parity slices and two hot-spares, potentially allowing for
|
|
5 HDD failures before data-loss occurs. However this is over-conservative for the pool that we'll be building out.
|
|
This pool won't be used for critical data, so we can afford to be a little more aggressive
|
|
with our dRAID setup.
|
|
That leaves us with 8/2/2, or 8/3/1.
|
|
|
|
These can both tolerate 4 sequential drive failures, however 8/2/2 has a slightly higher write and resilver speed
|
|
at the cost of tolerating less concurrent failures (2 at a time vs 3 at a time).
|
|
|
|
With how large spinning rust drives can get these days (18 - 20 TB each), two parity slices feels a little sketchy, so
|
|
we'll go with 8/3/1 as our template.
|
|
|
|
#### ZIL/L2ARC
|
|
We'll also need a ZIL and L2ARC device. Sun's old documentation recommends mirroring this device for resilience and
|
|
reliability reasons. That seems entirely reasonable, so let's do that, too.
|
|
We can also use these drives as a read cache to improve latency on popular data.
|
|
We'll want something with relatively high IOPS for this, as well as good random IO speeds.
|
|
In theory, we could go hog wild and get Intel Optane and never worry about it again, but that's...
|
|
prohibitively expensive to put it lightly.
|
|
|
|
Balancing cost and performance, 2 ✕ 500GB Seagate FireCuda 530s does the job well. Relatively high endurance,
|
|
good random IO performance, and not too expensive. Two of these sets us back 600 AUD.
|
|
|
|
#### Final Storage Configuration
|
|
Our final storage array consists of 12 ✕ 16 TB Seagate Exos Drives, and 2 ✕ 500GB Seagate FireCuda 530s.
|
|
With 8 data slices, 3 parity slices, and 1 hot spare slice, we should end up with approximately 110 TB of usable space.
|
|
|
|
{{< alert >}}
|
|
**(Ashe)** It is worth mentioning here that dRAID does not allow for variable stripe width due to how sequential
|
|
resilvering works, so compression ratios and real vs on-disk utilisation may suffer.
|
|
{{< /alert >}}
|
|
|
|
### The CPU
|
|
We're going to make an executive decision and go with an AMD EPYC CPU, because we've always wanted to use one.
|
|
That said, we have a few options:
|
|
|
|
| Model | Generation | {{< hover "Recommended Retail Price" >}}RRP{{< /hover >}} (USD) | Cores (threads) | Clock (GHz) | L3 Cache (MB) | TDP (W) |
|
|
| -------- |:----------:|:---------:|:---------------:|:-------------:|:-------------:|:-------:|
|
|
| 7272 | Rome | 625 | 12 (24) | 2.9 - 3.2 | 64 | 120 |
|
|
| 7302 | Rome | 978 | 16 (32) | 3.0 - 3.3 | 128 | 155 |
|
|
| **7352** | **Rome** | **1350** | **24 (48)** | **2.3 - 3.2** | **128** | **155** |
|
|
| 7402 | Rome | 1783 | 24 (48) | 2.8 - 3.35 | 128 | 180 |
|
|
| 7F72 | Rome | 2450 | 24 (48) | 3.5 - 3.9 | 192 | 240 |
|
|
| 7452 | Rome | 2025 | 32 (64) | 2.35 - 3.35 | 128 | 155 |
|
|
| 7453 | Milan | 1570 | 28 (56) | 2.75 - 3.45 | 64 | 225 |
|
|
| 7413 | Milan | 1825 | 24 (48) | 2.85 - 4.00 | 128 | 200 |
|
|
| 7513 | Milan | 2840 | 32 (64) | 2.6 - 3.65 | 128 | 200 |
|
|
|
|
The odd one out here is the `7F72`, which is a frequency-optimised model, designed for maximum performance per core
|
|
to get around per-core licensing issues in enterprise applications. While cool, it being nearly double the price
|
|
of the comparable `7352` puts it outside our budget for this particular build.
|
|
|
|
Again, balancing price and performance, we've decided to go with a `AMD EPYC 7352`, as 24/48 exceeds our spec, and doesn't break
|
|
the bank while doing so. We miss out on some of the performance of the Milan line, but that's acceptable here.
|
|
The SP3 socket also allows us to upgrade to Milan(-X) down the line if we need more performance (with a BIOS update).
|
|
|
|
Shipped to a friend, this sets us back ~1500 AUD.
|
|
|
|
### The Motherboard
|
|
|
|
With our CPU chosen, we need a motherboard that fulfills our purposes.
|
|
Of the readily-available options, we are looking for something with an
|
|
IPMI/BMC, and dual Ethernet interfaces onboard, as our data port requirement can be fulfilled by a PCIe network card.
|
|
|
|
8 SATA ports under one SATA controller would be nice, as it makes configuring passthrough for ZFS easier, but is not
|
|
essential.
|
|
|
|
The `AsRock Rack ROMED8U-2T` serves our purposes perfectly:
|
|
- [X] 8 ✕ DIMM slots
|
|
- [X] 8 ✕ SATA (2 ✕ mini-SAS)
|
|
- [X] Some amount of PCIe Slots (3)
|
|
- [X] 2 ✕ 10GbE + 1 ✕ IPMI
|
|
|
|
We don't *need* 10GbE for our management network, but this allows us to build this out with 2 ✕ 10GbE for data, and
|
|
upgrade to 2 ✕ 40GbE later, which may be what we end up doing. Notably, this leaves us without separate management and
|
|
data interfaces.
|
|
|
|
> **(Selene)** Is that a problem?
|
|
>
|
|
> **(Ashe)** Not really, no. It makes securing the network *slightly* trickier as we can't airgap the management network,
|
|
> but it's not the end of the world. We intend to get a PCIe Ethernet card at some point anyway, so this'll
|
|
> be temporary.
|
|
>
|
|
> **(Octavia)** Famous last words.
|
|
|
|
New from Newegg, this sets us back ~1100 AUD.
|
|
|
|
### The RAM
|
|
There are two ways we can estimate how much RAM we'll need.
|
|
- Total RAM required by all planned VMs
|
|
- RAM per vCPU
|
|
|
|
#### RAM by VM
|
|
|
|
1. Foundry Server
|
|
* 8GB will be plenty for this
|
|
2. Chibisafe
|
|
* Chibisafe prides itself on running lean, so we should be able to go down to as low as 1 or 2 GB
|
|
3. ZFS Storage Cluster
|
|
* The extremely conservative guideline (as published by Sun back in the day) is 1GB RAM per TB of storage.
|
|
This guideline was published with the idea that at this level you should *never* encounter any bottlenecks or issues.
|
|
* We do not need such a strict performance guarantee.
|
|
* We should be able to halve or even quarter this and not encounter bottlenecks.
|
|
* We'll initially provision 32GB, and adjust as necessary.
|
|
4. Modded Minecraft Server
|
|
* 16 - 32 GB is the rough ballpark for good performance with 8 - 16 players.
|
|
* This is likely overkill for our server, so we can dial it back to 12GB with some GC tuning on the server end[^1].
|
|
|
|
[^1]: GC tuning for Minecraft is its own beast, and one we could probably write a decent chunk about. Maybe a topic for
|
|
a future blog post?
|
|
|
|
So that totals up to 8 + 2 + 32 + 12 = 54 GB.
|
|
|
|
We want to allow room for growth and for friends to start their own VMs, so the next logical stepping stones are 64 or 128 GB.
|
|
|
|
#### Total Required RAM
|
|
We have 48 vCPUs in our current setup (with no overcommit, but more on that in the next blog post).
|
|
Very broadly, most environments that we've had exposure to allocate approximately 4GB/vCPU, and adjust based on how
|
|
CPU-hungry or memory-hungry a particular workload is.
|
|
Since we're expecting mixed workloads (from friends), we'll follow that same guideline.
|
|
We'll need at least 4 ✕ 48 = 192GB. With 8 slots on the mobo,
|
|
that means we'll have to use 32GB modules (or 64GB modules if we feel like going overboard).
|
|
|
|
Our motherboard comes with 8 RAM slots and the Rome EPYC CPUs support octo-channel RAM, so we'd get
|
|
the best performance from fully populating the RAM slots.
|
|
|
|
This however makes upgrading painful, so we're going to go with 4 ✕ 32GB @ 2400MHz for 128GB total for now, which sets
|
|
us back ~1000 AUD.
|
|
|
|
## The Damage
|
|
All up, we've settled on:
|
|
| Component | Selected Part | Cost (AUD) | Running Total (AUD) |
|
|
|:--------------:|:----------------------------------------------------:|:---------------:|:-------------------:|
|
|
| 3U Server Case | Who Even Knows? | FREE! | 0 |
|
|
| PSU | SilverStone 750W Platinum <br /> (SST-SX750-PT) | FREE! | 0 |
|
|
| CPU | AMD EPYC 7352 <br /> (24 core / 48 thread) | 1500 | 1500 |
|
|
| RAM | 4 ✕ 32GB@2400MHz <br /> (M393A4K40CB1-CRC) | 4 ✕ 250 = 1000 | 2500 |
|
|
| Mass Storage | 12 ✕ 16TB Seagate Exos <br /> (1ST16000NM001G) | 12 ✕ 400 = 4800 | 7300 |
|
|
| ZIL/L2ARC SSDs | 2 ✕ 500GB Seagate FireCuda 530 <br /> (ZP500GM3A013) | 2 ✕ 300 = 600 | 7900 |
|
|
| Motherboard | AsRock Rack ROMED8U-2T | 1100 | 9000 |
|
|
|
|
So just shy of 10,000 AUD. It's a hefty price tag, but worth it in our opinion.
|
|
|
|
## Next Time!
|
|
|
|
With all that decided upon, ordered, all we have to do now is wait for it to arrive.
|
|
|
|
Join us again next time for the actual build log!
|
|
|
|
> **(Doll)** This one is super excited to see Miss' build the nyoomy server!
|