diff --git a/assets/img/rin.png b/assets/img/rin.png new file mode 100644 index 0000000..58d6ad9 Binary files /dev/null and b/assets/img/rin.png differ diff --git a/config/_default/languages.en.yaml b/config/_default/languages.en.yaml index a92e035..21598b4 100644 --- a/config/_default/languages.en.yaml +++ b/config/_default/languages.en.yaml @@ -9,13 +9,13 @@ title: "Rin" # logo = "img/logo.png" description: "As in 凛冽时雨" copyright: "Copyright 2022 Rin (Tamara Vassileva)
All opinions herein are our own -and not those of employers past, present, or potential." +and not those of employers past, present, or potential.
Icon couresty of [https://picrew.me/image_maker/41329/](https://picrew.me/image_maker/41329/)" dateFormat: "Mon Jan 2 2006" author: name: "Rin" - image: "img/blowfish_logo.png" + image: "img/rin.png" headline: "As in 凛冽时雨" bio: "System that mostly works on TCP/IP networks and automation,
but loves FP and Category Theory" links: diff --git a/content/posts/server-build.md b/content/posts/server-build.md new file mode 100644 index 0000000..3245f6f --- /dev/null +++ b/content/posts/server-build.md @@ -0,0 +1,229 @@ +--- +title: "Our Home Server Build-Out Log Part 1: Speccing It Out" +date: 2022-10-31T17:00:00+11:00 +draft: false +showSummary: true +summary: "We've wanted to build a home lab for a long time. Here's how it went." +--- + +# Part One: Specification + +## Background + +We've wanted to build a home lab for a long, long time, but never got around to it. +**(Selene)** However, a certain someone was complaining about disaster recovery and migrating workloads to us, +and that turned out to be just the push we needed to get this rolling. + +Since we also want to do more blogging and community contribution, why not make this the inaugural post on our new blog? + +**(Lorelai)** So, let's get this show on the road, shall we? + +## The Use-Case + +So what did we need from this server? After talking over with friends, the following use-cases came to mind: + - Storage Cluster (ZFS of some kind) + - Foundry VTT Server + - [chibisafe](https://github.com/chibisafe/chibisafe) Server + - Modded Minecraft Server ? + - Generic cloud host for friends + +## The Spec +We wanted this server to be the basis for a home lab that could later be expanded, so we want it to have a decent amount of oomph. +After deliberating on it for a while, we decided on the following: + +| Part | Spec | +| ----------- | ----------------------------------------------------------- | +| CPU | Min. 16 core/32 thread | +| Motherboard | Must support large amounts of RAM and have BMC/IPMI | +| RAM | Moderate speed, 128GB to start, more to be added | +| Storage | At least 32 TB usable, 2 redundant disks min., 1 hot spare | +| Networking | Min. 2 ✕ 1Gbps outbound, 2 ✕ 10Gbps internal | + +### But Why? + +**(Selene)** I'm no techie, but this all seems rather excessive for a home server. Especially compared to people we know. + +**(Ashe)** You're not wrong there, Selene. I could wax lyrical about how I want this server to last a long time and be +useful for many different things and so on and so forth, but if we're being honest, the main reason is "Tech shiny!" + +**(Tammy)** So the same reason we've always gotten high spec PCs, really. + + +## Fulfilling the Spec +Now that we have a spec, we need to figure out the best way to fulfill it. + +### The Case +This would normally be something to consider carefully, but a good friend was getting rid of a 3U server case with +12 3.5" bays, so we'll nick that off of him, and move on. He also donated us a few +[Arctic P12s](https://www.arctic.de/en/P12/ACFAN00118A) for cooling as they'll be quieter than the jet engines +that came with the case. + +### The PSU +We have a spare 850W 80+ Gold PSU lying around the house from an old PC, so we'll just repurpose that here. + +### The Storage +It's probably strange to start with the storage, but we'll need to know how much storage we have to spec out +the ZFS instance properly. + +#### The Goal +The goal is to have a scaleable template that can be expanded in sets without breaking the bank each time, +while maintaining favourable failure characteristics. We'll budget out 3000 AUD for each expansion. + +**(Doll)** That sounds like a lot, Miss... + +**(Ashe)** It's a good chunk, yes. But this should, ideally, only happen every few years, so we can amortise this +over the life of the storage. + +We have 12 3.5" slots in our 3U compute case, and most storage arrays are either 24 or 36 bay, so planning around +12 unit slices seems logical. + +If we round up and call it 3600 AUD per slice, we get ~300 AUD per HDD. +At time of writing, the best $/GB is 16TB Seagate Exos drives. We picked up 6 a year ago when exchange rates +weren't so miserable, and will be picking up another 6 to round out this cluster after the new year. +This overshoots our budget by about 25%, but there's little to be done about exchange rates. + +We can, however, expect these to fall in price over time, +so this will likely be within budget by the time we need another slice. + + +#### RAID Layout +[dRAID](https://openzfs.github.io/openzfs-docs/Basic%20Concepts/dRAID%20Howto.html) came to OpenZFS a while back, +and has improved resilver times compared to Z1/Z2/Z3, as well as allowing for much more granular redundancy design, +so we'll be using that. + +The Seagate Exos drives are specc'd at ~200MB/s peak write rate, which gives us a resilver time of just over 22 hours in +traditional Z1/Z2/Z3. And that's 22 hours of every other drive in the array getting hammered. This is the main reason +we're going to be using dRAID. Under dRAID, we specify a number of data, parity, and spare slices, which are distributed +amongst the physical drives. + +In our case, we want at least one, maybe two, hot-spare slices to make best use of dRAID. +This leaves us with 10 or 11 slices to play with. + +With 11 usable slices our best bet would probably be 8/3/1 (8 data, 3 parity, 1 hot-spare), where as with 10 usable slices, +we'd probably want to go for either 7/3/2, or 8/2/2. + +7/3/2 is certainly the safest set-up, as it gives us 3 parity slices and two hot-spares, potentially allowing for +5 HDD failures before data-loss occurs. However this is over-conservative for the pool that we'll be building out initially. +This pool won't be the only storage location for critical data, so we can afford to be a little more aggressive +with our dRAID setup, so we're left with 8/2/2, or 8/3/1. + +These can both tolerate 4 sequential drive failures, however 8/2/2 has a slightly higher write and resilver speed +at the cost of tolerating less concurrent failures (2 at a time vs 3 at a time). + +With how large spinning rust drives can get these days (18 - 20 TB each), two parity slices feels a little sketchy, so +we'll go with 8/3/1 as our template. + +#### ZIL/L2ARC +We'll also need a ZIL and L2ARC device, which we can use some SSDs for. +We can also use these drives as a read cache. The recommendation is two use two mirrored drives for this +for resilience reasons, so we'll do that too. We'll want something with relatively high IOPS for this, as well as +good random IO speeds. +In theory, we could go hog wild and get Intel Optane and never worry about it again, but that's... +prohibitively expensive to put it lightly. +Again, balancing cose and performance, 2 ✕ 500GB Seagate FireCuda 530s does the job well. Relatively high endurance, +good random IO performance, and not too expensive. Two of these sets us back 600 AUD. + +### The CPU +We're going to make an executive decision and go with an AMD Epyc CPU, because we've always wanted to use one. +That said, we have a few options: + +| Model | Generation | RRP (USD) | Cores (threads) | Clock (GHz) | L3 Cache (MB) | TDP (W) | +| ----- |:----------:|:---------:|:---------------:|:-----------:|:-------------:|:-------:| +| 7272 | Rome | 625 | 12 (24) | 2.9 - 3.2 | 64 | 120 | +| 7302 | Rome | 978 | 16 (32) | 3.0 - 3.3 | 128 | 155 | +| 7352 | Rome | 1350 | 24 (48) | 2.3 - 3.2 | 128 | 155 | +| 7402 | Rome | 1783 | 24 (48) | 2.8 - 3.35 | 128 | 180 | +| 7F72 | Rome | 2450 | 24 (48) | 3.5 - 3.9 | 192 | 240 | +| 7452 | Rome | 2025 | 32 (64) | 2.35 - 3.35 | 128 | 155 | +| 7453 | Milan | 1570 | 28 (56) | 2.75 - 3.45 | 64 | 225 | +| 7413 | Milan | 1825 | 24 (48) | 2.85 - 4.00 | 128 | 200 | +| 7513 | Milan | 2840 | 32 (64) | 2.6 - 3.65 | 128 | 200 | + +The odd one out here is the `7F72`, which is a frequency-optimised model, designed for maximum performance per core, +to get around per-core licensing issues in enterprise applications. While cool, it being nearly double the price +of the comparable `7352` puts it outside our budget for this particular build. + +Balancing Price and Performance, we've decided to go with a `AMD EPYC 7352`, as 24/48 exceeds our spec, and doesn't break +the bank while doing so. We miss out on some of the performance of the Milan line, but that's acceptable here. +The SP3 socket also allows us to upgrade to Milan(-X) down the line if we need more performance (with a BIOS update). + +Shipped to a friend, this sets us back ~1500 AUD. + +### The Motherboard + +With our CPU chosen, we need a motherboard that fulfills our purposes. +Of the options, we are looking for something with an +IPMI/BMC, and dual Ethernet interfaces onboard, as our 40GbE port requirement can be fulfilled by a PCIe network card. + +8 SATA ports under one SATA controller would be nice, as it makes configuring passthrough for ZFS easier, but is not +essential. + +The AsRock Rack ROMED8U-2T serves our purposes perfectly: + - [X] 8 ✕ DIMM slots + - [X] 11 ✕ SATA (2 ✕ mini-SAS) + - [X] Some amount of PCIe Slots (3) + - [X] 2 ✕ 10GbE + 1 ✕ IPMI + +New from Newegg, this sets us back ~1100 AUD. + +### The RAM +There are two ways we can estimate how much RAM we'll need. + - Total RAM required by all planned VMs + - RAM per vCPU + +#### RAM by VM + +1. Foundry Server + * 8GB will be plenty for this +2. Chibisafe + * Chibisafe prides itself on running slim, so we should be able to go down to even 1 or 2 GB +3. ZFS Storage Cluster + * The extremely conservative guideline (as published by Sun back in the day) is 1GB RAM per TB of storage. + This guideline was published with the idea that at the level you should *never* encounter any bottlenecks or issues. + * We do not need such a strict performance guarantee. + * We should be able to halve or even quarter this and not encounter bottlenecks. + * We'll initially provision 32GB, and adjust as necessary. +4. Modded Minecraft Server + * 16 - 32 GB is the rough ballpark for good performance with 8 - 16 players + * This is likely overkill for this, so we can dial it back to 12GB with some GC tuning on the server end + +So that totals up to 8 + 2 + 32 + 12 = 54 GB. + +We want to allow room for growth and for friends to start their own VMs, so the next logical stepping stones are 64 or 128 GB. + +#### Total Required RAM +We have 48 vCPUs in our current setup (with no overcommit, but more on that later). +Very broadly, most environments that we've had exposure to allocate approximately 4GB/vCPU, and adjust based on how +CPU-hungry or memory-hungry a particular workload is. +Since we're expecting mixed workloads (from friends), we'll follow the same guideline of 4GB RAM per vCPU, +so we'll need at least 4 ✕ 48 = 192GB. With 8 slots on the mobo, +that means we'll have to use 32GB modules (or 64GB modules if we feel like going overboard). + +Our motherboard comes with 8 RAM slots and the Rome Epyc CPUs support octo-channel RAM, so we'd get +the best performance from fully populating the RAM slots. + +This however makes upgrading painful, so we're going to go with 4 ✕ 32GB @ 2400MHz for 128GB total for now, which sets +us back ~1000 AUD. + +## The Damage +All up, we've spent: +| Part | Cost (AUD) | Running Total (AUD) | +|:--------------:|:---------------:|:-------------------:| +| 3U Server Case | FREE! | 0 | +| PSU | FREE! | 0 | +| CPU | 1500 | 1500 | +| RAM | 1000 | 2500 | +| Mass Storage | 12 ✕ 400 = 4800 | 7300 | +| ZIL/L2ARC SSDs | 2 ✕ 300 = 600 | 7900 | +| Motherboard | 1100 | 9000 | +| | | | + +So just shy of 10,000 AUD. It's a hefty price tag, but worth it in our opinion. + +## Next Time! + +With all that decided upon, ordered, all we have to do now is wait for it to arrive. + +Join us again next time for the actual build log! + +**(Doll)** This one is super excited to see Miss' build the nyoomy server!