Server Build Article + Icon
AtelierSnek
parent
68dd97d94b
commit
5ac156e497
Binary file not shown.
|
After Width: | Height: | Size: 386 KiB |
|
|
@ -9,13 +9,13 @@ title: "Rin"
|
|||
# logo = "img/logo.png"
|
||||
description: "As in 凛冽时雨"
|
||||
copyright: "Copyright 2022 Rin (Tamara Vassileva) <br /> All opinions herein are our own
|
||||
and not those of employers past, present, or potential."
|
||||
and not those of employers past, present, or potential. <br /> Icon couresty of [https://picrew.me/image_maker/41329/](https://picrew.me/image_maker/41329/)"
|
||||
|
||||
dateFormat: "Mon Jan 2 2006"
|
||||
|
||||
author:
|
||||
name: "Rin"
|
||||
image: "img/blowfish_logo.png"
|
||||
image: "img/rin.png"
|
||||
headline: "As in 凛冽时雨"
|
||||
bio: "System that mostly works on TCP/IP networks and automation,<br />but loves FP and Category Theory"
|
||||
links:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,229 @@
|
|||
---
|
||||
title: "Our Home Server Build-Out Log Part 1: Speccing It Out"
|
||||
date: 2022-10-31T17:00:00+11:00
|
||||
draft: false
|
||||
showSummary: true
|
||||
summary: "We've wanted to build a home lab for a long time. Here's how it went."
|
||||
---
|
||||
|
||||
# Part One: Specification
|
||||
|
||||
## Background
|
||||
|
||||
We've wanted to build a home lab for a long, long time, but never got around to it.
|
||||
**(Selene)** However, a certain someone was complaining about disaster recovery and migrating workloads to us,
|
||||
and that turned out to be just the push we needed to get this rolling.
|
||||
|
||||
Since we also want to do more blogging and community contribution, why not make this the inaugural post on our new blog?
|
||||
|
||||
**(Lorelai)** So, let's get this show on the road, shall we?
|
||||
|
||||
## The Use-Case
|
||||
|
||||
So what did we need from this server? After talking over with friends, the following use-cases came to mind:
|
||||
- Storage Cluster (ZFS of some kind)
|
||||
- Foundry VTT Server
|
||||
- [chibisafe](https://github.com/chibisafe/chibisafe) Server
|
||||
- Modded Minecraft Server ?
|
||||
- Generic cloud host for friends
|
||||
|
||||
## The Spec
|
||||
We wanted this server to be the basis for a home lab that could later be expanded, so we want it to have a decent amount of oomph.
|
||||
After deliberating on it for a while, we decided on the following:
|
||||
|
||||
| Part | Spec |
|
||||
| ----------- | ----------------------------------------------------------- |
|
||||
| CPU | Min. 16 core/32 thread |
|
||||
| Motherboard | Must support large amounts of RAM and have BMC/IPMI |
|
||||
| RAM | Moderate speed, 128GB to start, more to be added |
|
||||
| Storage | At least 32 TB usable, 2 redundant disks min., 1 hot spare |
|
||||
| Networking | Min. 2 ✕ 1Gbps outbound, 2 ✕ 10Gbps internal |
|
||||
|
||||
### But Why?
|
||||
|
||||
**(Selene)** I'm no techie, but this all seems rather excessive for a home server. Especially compared to people we know.
|
||||
|
||||
**(Ashe)** You're not wrong there, Selene. I could wax lyrical about how I want this server to last a long time and be
|
||||
useful for many different things and so on and so forth, but if we're being honest, the main reason is "Tech shiny!"
|
||||
|
||||
**(Tammy)** So the same reason we've always gotten high spec PCs, really.
|
||||
|
||||
|
||||
## Fulfilling the Spec
|
||||
Now that we have a spec, we need to figure out the best way to fulfill it.
|
||||
|
||||
### The Case
|
||||
This would normally be something to consider carefully, but a good friend was getting rid of a 3U server case with
|
||||
12 3.5" bays, so we'll nick that off of him, and move on. He also donated us a few
|
||||
[Arctic P12s](https://www.arctic.de/en/P12/ACFAN00118A) for cooling as they'll be quieter than the jet engines
|
||||
that came with the case.
|
||||
|
||||
### The PSU
|
||||
We have a spare 850W 80+ Gold PSU lying around the house from an old PC, so we'll just repurpose that here.
|
||||
|
||||
### The Storage
|
||||
It's probably strange to start with the storage, but we'll need to know how much storage we have to spec out
|
||||
the ZFS instance properly.
|
||||
|
||||
#### The Goal
|
||||
The goal is to have a scaleable template that can be expanded in sets without breaking the bank each time,
|
||||
while maintaining favourable failure characteristics. We'll budget out 3000 AUD for each expansion.
|
||||
|
||||
**(Doll)** That sounds like a lot, Miss...
|
||||
|
||||
**(Ashe)** It's a good chunk, yes. But this should, ideally, only happen every few years, so we can amortise this
|
||||
over the life of the storage.
|
||||
|
||||
We have 12 3.5" slots in our 3U compute case, and most storage arrays are either 24 or 36 bay, so planning around
|
||||
12 unit slices seems logical.
|
||||
|
||||
If we round up and call it 3600 AUD per slice, we get ~300 AUD per HDD.
|
||||
At time of writing, the best $/GB is 16TB Seagate Exos drives. We picked up 6 a year ago when exchange rates
|
||||
weren't so miserable, and will be picking up another 6 to round out this cluster after the new year.
|
||||
This overshoots our budget by about 25%, but there's little to be done about exchange rates.
|
||||
|
||||
We can, however, expect these to fall in price over time,
|
||||
so this will likely be within budget by the time we need another slice.
|
||||
|
||||
|
||||
#### RAID Layout
|
||||
[dRAID](https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html) came to OpenZFS a while back,
|
||||
and has improved resilver times compared to Z1/Z2/Z3, as well as allowing for much more granular redundancy design,
|
||||
so we'll be using that.
|
||||
|
||||
The Seagate Exos drives are specc'd at ~200MB/s peak write rate, which gives us a resilver time of just over 22 hours in
|
||||
traditional Z1/Z2/Z3. And that's 22 hours of every other drive in the array getting hammered. This is the main reason
|
||||
we're going to be using dRAID. Under dRAID, we specify a number of data, parity, and spare slices, which are distributed
|
||||
amongst the physical drives.
|
||||
|
||||
In our case, we want at least one, maybe two, hot-spare slices to make best use of dRAID.
|
||||
This leaves us with 10 or 11 slices to play with.
|
||||
|
||||
With 11 usable slices our best bet would probably be 8/3/1 (8 data, 3 parity, 1 hot-spare), where as with 10 usable slices,
|
||||
we'd probably want to go for either 7/3/2, or 8/2/2.
|
||||
|
||||
7/3/2 is certainly the safest set-up, as it gives us 3 parity slices and two hot-spares, potentially allowing for
|
||||
5 HDD failures before data-loss occurs. However this is over-conservative for the pool that we'll be building out initially.
|
||||
This pool won't be the only storage location for critical data, so we can afford to be a little more aggressive
|
||||
with our dRAID setup, so we're left with 8/2/2, or 8/3/1.
|
||||
|
||||
These can both tolerate 4 sequential drive failures, however 8/2/2 has a slightly higher write and resilver speed
|
||||
at the cost of tolerating less concurrent failures (2 at a time vs 3 at a time).
|
||||
|
||||
With how large spinning rust drives can get these days (18 - 20 TB each), two parity slices feels a little sketchy, so
|
||||
we'll go with 8/3/1 as our template.
|
||||
|
||||
#### ZIL/L2ARC
|
||||
We'll also need a ZIL and L2ARC device, which we can use some SSDs for.
|
||||
We can also use these drives as a read cache. The recommendation is two use two mirrored drives for this
|
||||
for resilience reasons, so we'll do that too. We'll want something with relatively high IOPS for this, as well as
|
||||
good random IO speeds.
|
||||
In theory, we could go hog wild and get Intel Optane and never worry about it again, but that's...
|
||||
prohibitively expensive to put it lightly.
|
||||
Again, balancing cose and performance, 2 ✕ 500GB Seagate FireCuda 530s does the job well. Relatively high endurance,
|
||||
good random IO performance, and not too expensive. Two of these sets us back 600 AUD.
|
||||
|
||||
### The CPU
|
||||
We're going to make an executive decision and go with an AMD Epyc CPU, because we've always wanted to use one.
|
||||
That said, we have a few options:
|
||||
|
||||
| Model | Generation | RRP (USD) | Cores (threads) | Clock (GHz) | L3 Cache (MB) | TDP (W) |
|
||||
| ----- |:----------:|:---------:|:---------------:|:-----------:|:-------------:|:-------:|
|
||||
| 7272 | Rome | 625 | 12 (24) | 2.9 - 3.2 | 64 | 120 |
|
||||
| 7302 | Rome | 978 | 16 (32) | 3.0 - 3.3 | 128 | 155 |
|
||||
| 7352 | Rome | 1350 | 24 (48) | 2.3 - 3.2 | 128 | 155 |
|
||||
| 7402 | Rome | 1783 | 24 (48) | 2.8 - 3.35 | 128 | 180 |
|
||||
| 7F72 | Rome | 2450 | 24 (48) | 3.5 - 3.9 | 192 | 240 |
|
||||
| 7452 | Rome | 2025 | 32 (64) | 2.35 - 3.35 | 128 | 155 |
|
||||
| 7453 | Milan | 1570 | 28 (56) | 2.75 - 3.45 | 64 | 225 |
|
||||
| 7413 | Milan | 1825 | 24 (48) | 2.85 - 4.00 | 128 | 200 |
|
||||
| 7513 | Milan | 2840 | 32 (64) | 2.6 - 3.65 | 128 | 200 |
|
||||
|
||||
The odd one out here is the `7F72`, which is a frequency-optimised model, designed for maximum performance per core,
|
||||
to get around per-core licensing issues in enterprise applications. While cool, it being nearly double the price
|
||||
of the comparable `7352` puts it outside our budget for this particular build.
|
||||
|
||||
Balancing Price and Performance, we've decided to go with a `AMD EPYC 7352`, as 24/48 exceeds our spec, and doesn't break
|
||||
the bank while doing so. We miss out on some of the performance of the Milan line, but that's acceptable here.
|
||||
The SP3 socket also allows us to upgrade to Milan(-X) down the line if we need more performance (with a BIOS update).
|
||||
|
||||
Shipped to a friend, this sets us back ~1500 AUD.
|
||||
|
||||
### The Motherboard
|
||||
|
||||
With our CPU chosen, we need a motherboard that fulfills our purposes.
|
||||
Of the options, we are looking for something with an
|
||||
IPMI/BMC, and dual Ethernet interfaces onboard, as our 40GbE port requirement can be fulfilled by a PCIe network card.
|
||||
|
||||
8 SATA ports under one SATA controller would be nice, as it makes configuring passthrough for ZFS easier, but is not
|
||||
essential.
|
||||
|
||||
The AsRock Rack ROMED8U-2T serves our purposes perfectly:
|
||||
- [X] 8 ✕ DIMM slots
|
||||
- [X] 11 ✕ SATA (2 ✕ mini-SAS)
|
||||
- [X] Some amount of PCIe Slots (3)
|
||||
- [X] 2 ✕ 10GbE + 1 ✕ IPMI
|
||||
|
||||
New from Newegg, this sets us back ~1100 AUD.
|
||||
|
||||
### The RAM
|
||||
There are two ways we can estimate how much RAM we'll need.
|
||||
- Total RAM required by all planned VMs
|
||||
- RAM per vCPU
|
||||
|
||||
#### RAM by VM
|
||||
|
||||
1. Foundry Server
|
||||
* 8GB will be plenty for this
|
||||
2. Chibisafe
|
||||
* Chibisafe prides itself on running slim, so we should be able to go down to even 1 or 2 GB
|
||||
3. ZFS Storage Cluster
|
||||
* The extremely conservative guideline (as published by Sun back in the day) is 1GB RAM per TB of storage.
|
||||
This guideline was published with the idea that at the level you should *never* encounter any bottlenecks or issues.
|
||||
* We do not need such a strict performance guarantee.
|
||||
* We should be able to halve or even quarter this and not encounter bottlenecks.
|
||||
* We'll initially provision 32GB, and adjust as necessary.
|
||||
4. Modded Minecraft Server
|
||||
* 16 - 32 GB is the rough ballpark for good performance with 8 - 16 players
|
||||
* This is likely overkill for this, so we can dial it back to 12GB with some GC tuning on the server end
|
||||
|
||||
So that totals up to 8 + 2 + 32 + 12 = 54 GB.
|
||||
|
||||
We want to allow room for growth and for friends to start their own VMs, so the next logical stepping stones are 64 or 128 GB.
|
||||
|
||||
#### Total Required RAM
|
||||
We have 48 vCPUs in our current setup (with no overcommit, but more on that later).
|
||||
Very broadly, most environments that we've had exposure to allocate approximately 4GB/vCPU, and adjust based on how
|
||||
CPU-hungry or memory-hungry a particular workload is.
|
||||
Since we're expecting mixed workloads (from friends), we'll follow the same guideline of 4GB RAM per vCPU,
|
||||
so we'll need at least 4 ✕ 48 = 192GB. With 8 slots on the mobo,
|
||||
that means we'll have to use 32GB modules (or 64GB modules if we feel like going overboard).
|
||||
|
||||
Our motherboard comes with 8 RAM slots and the Rome Epyc CPUs support octo-channel RAM, so we'd get
|
||||
the best performance from fully populating the RAM slots.
|
||||
|
||||
This however makes upgrading painful, so we're going to go with 4 ✕ 32GB @ 2400MHz for 128GB total for now, which sets
|
||||
us back ~1000 AUD.
|
||||
|
||||
## The Damage
|
||||
All up, we've spent:
|
||||
| Part | Cost (AUD) | Running Total (AUD) |
|
||||
|:--------------:|:---------------:|:-------------------:|
|
||||
| 3U Server Case | FREE! | 0 |
|
||||
| PSU | FREE! | 0 |
|
||||
| CPU | 1500 | 1500 |
|
||||
| RAM | 1000 | 2500 |
|
||||
| Mass Storage | 12 ✕ 400 = 4800 | 7300 |
|
||||
| ZIL/L2ARC SSDs | 2 ✕ 300 = 600 | 7900 |
|
||||
| Motherboard | 1100 | 9000 |
|
||||
| | | |
|
||||
|
||||
So just shy of 10,000 AUD. It's a hefty price tag, but worth it in our opinion.
|
||||
|
||||
## Next Time!
|
||||
|
||||
With all that decided upon, ordered, all we have to do now is wait for it to arrive.
|
||||
|
||||
Join us again next time for the actual build log!
|
||||
|
||||
**(Doll)** This one is super excited to see Miss' build the nyoomy server!
|
||||
Loading…
Reference in New Issue