Description

Time for a twist An actual custom 4U server on pcpartpicker for a change!

A build with a bunch of new hard drives for a new NAS/local compute server with plenty of power for cpu heavy tasks and virtualization.

This will finally allow me to get rid of the last Windows computer that I personally use, which was my old NAS. Been wanting to ditch it for some form of Linux for a while now, but couldn't because of the implications it had on data migration.

Focused around Proxmox as a hypervisor, utilizing ZFS (on Linux) for the main storage pool with an accelerated ZIL/L2ARC utilizing the m.2 NVMe SSDs and a separate SATA SSD for boot.

The ZFS pool will be served to the LAN over NFS for networked storage purposes, aside from also serving as the storage for all local VMs.

Proxmox has an easy configuration to setup/use a zfs pool as your main root/boot partition, however I want to specifically avoid this for various reasons (it's generally not a good idea to use your primary data store as dual-duty boot drive). Because of this, I will use a simple sata ssd as my proxmox boot drive, separated from the rest of the storage in the system. This could be avoided with a hypervisor that runs in RAM, booted from some sort of read-only live media (i.e. usb), however proxmox doesn't really support this (at least not well). Alternative hypervisors like ESXi or SmartOS do support this and wouldn't need a SATA disk dedicated for installation/booting. I was originally planning on using SmartOS because of the superb ZFS support, however the AMD-V virtualization extension support for KVM is experimental at best on SmartOS, so I fell back to proxmox.

8 of the 10TB HDDs will be put into a RAID-Z2 VDEV for the ZFS pool as well as 8 of the 4TB HDDs into a separate RAID-Z2 VDEV for the same pool. A single 10TB hard drive and 4TB hard drive are included as cold spares for hard drive failure replacement. I included the 4TB hard drives because I already had many of them laying around, even though the $/GB wouldn't really make sense if buying them new. This pool will be used as the primary data store for the local server VMs as well as over the network with NFS. This configuration should yield about ~70TiB of usable storage after the pool is created, although from a practical perspective, I will never really want to use more than ~56TiB in efforts to maintain at least 20% free space to ensure high performance. (Data integrity + performance has a high cost with storage). If I ever need to use more than 56TiB of storage, I'll simply add more drives for a new VDEV to add to the pool. This configuration also allows me to lose any 2 HDDs from each of the 2 VDEVs (so two 10TB and two 4TB drives) at once, and remain operational while I replace them (this ZFS configuration is equivalent to a RAID-60 in terms of parity/failure).

Initially with the 2 LSI cards (acting as HBAs in IT mode), I will only enable 16 (of the 24) HDD ports on the case, but this can be expanded by simply adding another 8-port sata HBA in the future.

The 2 NVMe SSDs will be utilized so that each of them will have a 32GB partition which will be used for a mirrored ZIL, and the remaining space on each will be used as a striped L2ARC cache (~400GB). Partitioning drives to share ZIL/L2ARC tasks is not ideal, however the NVMe SSDs will provide more than sufficient bandwidth/iops. I wanted a mirrored ZIL with quick SSDs, as well as a nice L2ARC cache, and rather than buying 3+ SATA SSDs without partitioning them for this job, I figure 2 high performance NVMe SSDs will make up for the fact that they are being exposed to the ZFS pool as partitions rather than whole disks.

For future upgrades, I'm considering adding a 10Gb/s NIC (once I swap all my LAN over to 10Gb/s gear), additional HDDs for a new VDEV, and another LSI card would be necessary to accompany any new drives. Adding more RAM in the future is a possibility if 64GB doesn't turn out to be enough in the long-run (ZFS is definitely a memory hog).

Notes: I recognize that I am not using ECC RAM despite using ZFS, and am fully aware of the risks. There really wasn't very much availability for unregistered 16GB DDR4 DIMMS, and I'm ok with the risk that it brings by not having ECC memory (plus the faster memory that I chose will be beneficial to the speed of the threadripper CPU). I also recognize that even with 64GB of RAM, for what I'm using it for, this could still be considered on the lower side (both memory hungry ZFS plus virtualization). I am fully aware of this, and if it becomes a problem with my particular workloads, I can easily add more RAM in the future.

Comments are encouraged.

Comments

  • 9 months ago
  • 15 points

Amazing! Really neat to see a system that makes a good pair out of the Threadripper and and the GT 710 :)

  • 9 months ago
  • 2 points

lol. Not like the GT 710 would be a bottleneck, or anything...... (sarcasm) +1 from me too.

  • 9 months ago
  • 3 points

It's a graphics output. shrugs

  • 9 months ago
  • 0 points

Yeah, don't you need that in this gaming rig? Look at all the games it can store! /s

  • 9 months ago
  • 14 points

Yup, it’s confirmed I think OP just started a hard drive shortage by accident

  • 9 months ago
  • 9 points

I don't even know what half this means but looks cool

  • 9 months ago
  • 8 points

You may not like it, but this is what peak performance.

  • 9 months ago
  • 4 points

looks like

  • 9 months ago
  • 4 points

my ocd thanks you.

  • 9 months ago
  • 5 points

Best cpu + video card pairing ever

  • 9 months ago
  • 7 points

It makes sense for his uses.

  • 9 months ago
  • 3 points

I know just being dumb. This seems like an LTT video tho, just the way it’s together.

  • 9 months ago
  • 1 point

lol

  • 9 months ago
  • 5 points

126TB AM I READING THIS RIGHT!?!?

Linus would be proud

(づ ʘ ╭͜ʖ╮ ʘ )づ

  • 9 months ago
  • 8 points

no, he would be laughing in his petabyte of spinning storage, and I think he has over 100TB in fast SSD storage too

  • 9 months ago
  • 2 points

yea yor right lol

  • 9 months ago
  • 1 point

Old news, I did this years ago for a Hollywood big-shot studio. Had to be big, so big, it had to be never been done before ....

Some idiot dropped GlusterFS ... Well, sheet. Money can fix anything you want anyway. Ever server is quad bonded by Intel's 10GbeE Server cards, all 9K Jumbo Frames, long story short, this puppy could drop a 4/8/12/16K RAW RED-Camera reels all day. (Google it).

Thoses things would let macpro's light up in fire. https://imgur.com/BT4IlyO

  • 9 months ago
  • 2 points

how much storage?

  • 9 months ago
  • 4 points

Very cool. Nice change of pace. Thanks for sharing!

  • 9 months ago
  • 4 points

Not seeing a 10Gb Ethernet interface; getting data in/out of that thing will take days over a gigabit link.

I hope your decision to forego ECC RAM doesn't bite you.

If data integrity was important enough to set up ZFS with RAID-Z2, and you're prepared to spend north of six grand putting it together, then you might also want to think about adding an LTO-6 or better tape drive for backups.

Otherwise, it's a very impressive build.

  • 9 months ago
  • 2 points

Yup, I'm planning on putting in 10Gb once I upgrade my local network to support it. my home network just isn't 10Gb yet, so I didn't pre-emptively get a 10Gb card.

Not too worried about tape drives for backup, any actually critical data is backed up in more than just this NAS, losing data on this would be annoying, but not catastrophic (again, this isn't a prod system, just a home server). The only reason I went RAID-Z2 is because having another drive die while replacing/resilvering is a very real possibility with big drives like these 10TB, especially since many are made in the same batch, so they can likely fail near each other. I'm not too concerned about the data integrity, and I would never run a system like this in prod.

I also hope the ECC RAM doesn't bite me. I would've preferred ECC, but I was (clearly) willing to take the speed trade-off. But like I mentioned, good unbuffered ECC DDR4 just doesn't really exist.

  • 9 months ago
  • 1 point

BTW, where are you keeping this thing? That many spinning disks and fans can't possibly be at all quiet.

Blatant Self-Aggrandizement: I built a new FreeNAS box for myself earlier this year. Just one RAID-Z2 vdev, but 16TB should keep me happy for a while.

  • 9 months ago
  • 1 point

That's the point of all the noctua fans. With every fan in the system running at max speed 100% of the time (my current config), the fans are still much quieter than the disks, which really aren't that loud. I did this very intentionally so I could have it out without being a nuisance. My old NAS was actually much louder than this one, despite only having 3 fans, and 7 drives.

This thing just sits next to my TV for now, I'm planning on getting a server rack when I upgrade my networking and get proper rack mounted switches.

  • 9 months ago
  • 3 points

This (actually very good) build has made me realize two things: 1. PCPP really needs to aggregate big quantities of the same item. And 2. Noctua has another fugly color scheme? Damn.

[comment deleted]
  • 9 months ago
  • 2 points

Yeah... but you're strange. ;)

[comment deleted]
  • 9 months ago
  • 2 points

looks crazy man +1

  • 9 months ago
  • 2 points

their been some debate on ecc ram compare to the raid card set up with correct software.

  • 9 months ago
  • 2 points

Would anybody like to explain to me what this is.

  • 9 months ago
  • 1 point

A computer that runs many virtual computers with lots of storage.

  • 9 months ago
  • 2 points

I must say its the little things that count like how your carpet is clean. But overall good build

  • 9 months ago
  • 2 points

have fun downloading a 4K game of the actual earth.

  • 9 months ago
  • 1 point

Nice! I was thinking about doing something like this with Threadripper for a NAS but I think I'm gonna go for an Epyc. Proxmox looks like a REALLY nice solution for the price.

  • 9 months ago
  • 2 points

Yup, Epyc would be ideal (support for multi CPU as well as buffered ECC support (Threadripper supports unbuffered ECC, but good luck finding good unbuffered DDR4)), but at like 2-4x the price of the equivalent threadripper CPU, not to mention the more expensive motherboard as well. If you've got a lot more money to spend, Epyc is the way to go if you don't want to drop $10k+ on the equivalent Intel Xeon CPUs.

Proxmox is awesome. Even though I'm a linux guy, I would have preferred SmartOS due to the amazing ZFS support on illumos, but SmartOS is basically Intel only due to the lack of AMD virtualization extension support. With that said, being able to run lxc containers on proxmox is great due to the insanely low overhead (you can run ubuntu server with like 50MB of RAM). I just wish ZFS on Linux was a tad farther along (although it certainly isn't bad).

  • 9 months ago
  • 1 point

I'm going with an ATX Supermicro H11SSL-NC. It's actually about $470, so not as expensive as some of the more extreme x399 boards. I think it's kind of worth it since it has onboard graphics, ipmi + kvm over lan, 8 sata, and 8 sas with the onboard controller. I wanted to go with the 8 core 7251 since it's only about $530, but it's down on cache and only guaranteed to work with 2400 RAM. Not sure if it's worth it to spend like $350 more for double the cores/L3cache and 2666 support.

  • 9 months ago
  • 1 point

Yeah, with those features, the extra mobo price is definitely worth it. The price on epyc was definitely the tipping point for me though. They do have a lot more cache, but, for example, that 7251 is currently selling for more than a 1950x (you can commonly find 1950x's at or below $500 now that the 2000 series threadripper are out), but has half of the cores, and a reasonably slower clockrate. So even with 2 of them, making them over 2x the cost, the 1950x will still perform better in raw compute.

As for ram speeds, I know zen really likes faster ram, especially the larger threadripper/epyc processors with multiple CCXs and infinity fabric. With that said, I'm not sure how much more performance you'll get out of +266mhz on the RAM, BUT with threadripper you can easily overclock ram much past the technically support 2667mhz (most people report using an XMP profile on 3200mhz RAM generally just works out of the box after selecting it in the bios), which actually does give a sizeable boost to performance.

That's why I went with the higher clocked (non-ECC) ram for this build. Threadripper can actually utilize it.

  • 9 months ago
  • 1 point

Yeah. The price for Threadrippers are unbeatable rn. I got my dad a 1920X for 390$ AFTER TAX. Thats not all I still had a 100$ gift card lying around and paid only 290 after tax. He only really uses photoshop and light 3d rendering so the 1920X will be a complete overkill but I just couldn't pass up the opportunity to build my first HEDT rig for basically the same price as a consumer desktop.

  • 9 months ago
  • 1 point

Awesome username p.s. amazing build too +1

  • 9 months ago
  • 1 point

congrats on the feature

  • 9 months ago
  • 1 point

This is way-over-my-head-next-level stuff. The only thing I know for sure is that you managed to get all of Noctua's ugliest fans into the same build at one time. +1 :)

  • 9 months ago
  • 1 point

just out of curiosity what do you need 126TB of storage for?

  • 9 months ago
  • 1 point

The bulk of it is media and general data archiving. Although worth noting it ends up being about ~70TiB usable after all redundancy and whatnot is calculated. Reliable data storage is not cheap, unfortunately.

  • 9 months ago
  • 1 point

That's one heck of a NAS. What do you need all that storage for? Are you editing 8K RAW footage? I heard that takes lots of space.

  • 9 months ago
  • 1 point

No but I hoard/archive data.

  • 9 months ago
  • 1 point

I saw "126GB NAS" and literally said aloud "Holy ****..."

Was so stunned that I even typed "allowed" the first time lol.

  • 9 months ago
  • 1 point

I dream of building myself a rig like this someday, in a future home, my own rack mount and switches and router...

  • 9 months ago
  • 1 point

WITNESS THE TRUE AND ULTIMATE POWER OF A PC GAMER

  • 9 months ago
  • 1 point

Great work. I am interested in building a NAS myself. I have a leftover 6700k (w. 32gb ram) I was going to use for it, do you think that CPU/RAM combo is too underpowered?

Thanks

  • 9 months ago
  • 2 points

Depends what you're using it for, and what storage solution you go with. If it's purely a NAS, and doesn't do dual duty as a virtualization server (like mine), then that's more than enough cpu/RAM to do whatever you want for storage. FreeNAS is a nice OS with a web interface that can help you build ZFS Pools, which is basically a really nice RAID.

If you are trying to do VMs as well, you can, but you'll definitely want more RAM if you do ZFS, and even if you don't, with the 6700k you probably won't want to spin up more than maybe 4-ish VMs or so total. Still totally do-able though.

  • 9 months ago
  • 1 point

Nice. Thanks for that information.

  • 9 months ago
  • 1 point

Nice, reminds of Linus' crazy server days.

  • 9 months ago
  • 1 point

Why not use BTRFS? I'm not familiar with Proxmox but I know of other hypervisor OS's that support it, and if your main purpose is storage why even bother with hypervisor availability? In production all hypervisors are CPU/RAM heavy with some very fast local storage for cashing, you seam to frogo the RAM for storage, were you planing on spinning up one vm to share all that space? With ZFS you would be limited in your VM ram allocation, I'm just curious what your plan with this was. Perhaps you were going to create iSCSI or some other network share from the host hypervisor and use it as networked storage within the VM(to reshare it out?) but that would create unneeded network traffic. Nice build, tho, I'm just not sure of the Hpervisor/Storage server approach your going for.

  • 9 months ago
  • 1 point

In regards to BTRFS: As far as I'm concerned, BTRFS is a half-baked filesystem solution for the people on linux who were butt-hurt about the CDDL licensing of OpenZFS, and wanted a GPL-compliant file system equivalent (although BTRFS is no where near feature parity or stability of ZFS). There have been many arguments over the years whether or not CDDL is truly compatible with GPLv2 or not, and although Stallman disagrees, there are many people (including lawyers), who say it's probably fine, and people continue to work on ZFS on Linux regardless, with no one trying to stop them yet. I don't think ZFS will ever be shipped as part of mainline linux because of the licensing, but that doesn't stop it from being more supported, stable, and suitable for real-world use vs BTRFS. I think you'd be hard-pressed to find almost ANYONE running BTRFS in production because of how experimental/unsupported it is. If you do some research, you'll see just how embryonic BTRFS really is (for a filesystem. Losing data is the worst type of failure, and it still happens with BTRFS deployments). ZFS is tried-and-true, and really the only open source local storage pooling solution that is decent. I could go on talking about it for forever, but I'll save us both some time...

Why have all my storage in my hypervisor? Well that's because it's my home network, and it reduces network/hardware overhead, since I have no need for a dedicated SAN with my only goal being to share storage with the LAN+VMs. You mentioned an iSCSI box that I shared to the hypervisor, and I did consider that at one point, but as you mention, it's just unnecessary network overhead, on top of needing an entirely extra box to host the storage. I would rather over-provision a single server and have everything in one. Proxmox is simply some vm-management software slapped on top of debian, so even though I call it a hypervisor, it really is a full linux OS just running some management software, and less of a true hypervisor. I realize, and am OK with the fact that ZFS uses a lot of resources that will take away what's available from all my VMs, but that's fine with me. Also note like 90% of my VMs are actually linux containers (LXC). This combined with the fact that I don't really run a lot of RAM-heavy software like databases, redis, etc (with the exception being ZFS, of course), means that my virtualization workload is much, much, much less RAM intensive than a typical hypervisor workload. Most of my VMs (containers) use less than 200MB of RAM when running.

The other thing about primarily containers for virtualization, is it means that these VMs get native access to the ZFS filesystem/pool without ANY extra overhead. They can access data from the pool exactly like the hypervisor, with the shared kernel (NO networking stack involved). This means I can have ZFS volumes shared between VMs with literally 0 overhead (where traditionally you'd need some sort of SAN and a very good network to support this, on top of simply the general overhead and latency it takes to run all that storage over wires over a network). This allows me to take the 'micro-services' approach to virtualization, where I can basically dedicate an entire VM (container) to each major application I want to run (so my network fileshare software can run independent of my media server which can run independent of everything else, despite sharing a lot of the same volumes/files). This is great from a security perspective, and allows me to achieve a really nice isolation of services without a single massive VM, and effectively 0 extra overhead from a Storage/CPU perspective, and only a tiny bit of extra overhead from a RAM perspective (something in the realm of 50-150MB per container).

Any KVM-based VMs will still need to use something like NFS to talk back with the hypervisor to get access to any shared storage (aside from what is directly allocated to them as a VM), which has a lot of overhead. If I was primarily using Windows, or KVM virtual machines, this model of having the storage on my hypervisor wouldn't make much sense, because even though network traffic doesn't have to go over a wire to access the shared storage, it's still going through a networking stack on essentially 2 different machines (VM+Hypervisor) ON TOP OF the regular kernel calls to access that data. Fortunately, I keep these VMs that require emulated kernels to a minimum, and I simply don't share my bulk storage with them (they just get allocated whatever they need as their primary block volume, and don't interact with any other storage).

tl;dr: BTRFS is kind of a joke tbh, ZFS is battle-tested, stable, and has more features. With my primary virtualization being containers (not emulated kernels with KVM), my RAM usage is much lower than a typical virtualization workload, hence the low amounts of RAM. This is coupled with the fact that containers get native access to the ZFS pool with no overhead, so if I have one big storage pool on my hypervisor, I can share parts of it between VMs (containers) with literally 0 extra overhead, so having something like a SAN with iSCSI would be detrimental to performance, even if it didn't have any network latency, simply due to the fact that all storage requests must go through networking stacks.

  • 9 months ago
  • 1 point

IMO you're a bit harsh on BTRFS, but I won't argue the conclusion which is that ZFS is better. I don't deal with ZFS much any more, unfortunately, because our customers are mostly heading towards linux if they aren't there already, and on linux we point them at XFS or less often ext4 because they run what's in the enterprise distro they bought. btrfs's fatal flaw for my purposes is that O_SYNC or fsync is dog slow, or was; it's been a year or two since I last tried it but there's nothing compelling there.

Meanwhile, when I spin up the Sun ultra 60 in the back, which happens 2-3 times a year, it comes up very reluctantly and there are nearly always one or two drives which need a cable replug or other slap upside the head to get running. ZFS just waits until I get it working, most filesystems would have a conniption and fall over dead. I do wish that a license reconciliation could be reached.

  • 9 months ago
  • 1 point

Reading back on my post, I think I definitely was a little harsh. I guess I shouldn't have called it half-baked, but I stand by my point that BTRFS is absolutely not production ready. I have been told many accounts from colleagues that tried deploying BTRFS with some smaller customers, only to have a filesystem bug destroy data. You only have to experience that once to generally never want to use it again. This sentiment is backed up in many places including RHEL's decision to deprecate BTRFS support. And of course, that's still all putting aside the fact that the feature-set simply isn't there with BTRFS to be really compelling in the first place.

It would certainly be ideal if OpenZFS were licensed better, but as it is, there is no precedent that anyone really cares about ZFS + Linux from any sort of legal perspective. Reminder that Canonical actually ships ubuntu with the ZFS kernel module pre-installed by default, and some lawyers have backed this up saying that from a legal perspective, it's within the rights of the licensing of both linux and ZFS.

For something like the bootdrive of a server, something like ext4 or XFS is fine, and slightly better when used in tandem with LVM, but for big pools of disks, you really want a filesystem that was designed for hardware failure while managing many disks at a time. Features like copy-on-write (which help enable things like native snap-shotting), can also be a huge plus, and huge money-saver when working that into your backup workflow, where you almost CAN'T afford to NOT use it.

  • 8 months ago
  • 1 point

Thanks for the additional information, when you mentioned you were using VM's I understood that to mean independent VM's not containers. For what you talking about containers make more sense. I'm not familiar enough with ZFS but maybe you can answer a question for me. Hardware raid or no? For servers with dedicated raid cards, that is. Thoughts on using ZFS and the hardware raid or not, same question with software raid.

Thanks.

  • 8 months ago
  • 1 point

It really depends what you're trying to do. A common argument for hardware raid in the past is that it offloaded calculations from your CPU onto a dedicated controller, although nowadays, pretty much any form of software raid is cheap enough (from a computational perspective), that it really doesn't matter anymore.

The main thing about hardware raid that I dislike is your stuck to traditional raid 0/1/5/6 or some combination. With this, it's impossible for you to change the raid size/disk configuration after the pool has been created. For that reason, I'm not too fond of traditional raid. If that doesn't bug you, traditional raid is fine, although still lacking in features compared to something like ZFS.

As for software raid, I'm not particularly fond of OS-dependent RAID solutions (where you create the raid inside the OS), because that locks you into that OS. Something like Intel's RST where you can create a RAID before boot is a really great, cheap, and easy solution, as long as you have a compatible Intel CPU (AMD doesn't offer anything like this at the moment).

With all of that said, if you have the hardware (namely the RAM), and the willingness to learn a little bit about storage administration, I definitely prefer ZFS over any RAID setup because of its features, performance, reliability, and configurability. ZFS is one of the most popular filesystems in production servers for a good reason. The only thing about ZFS is you have to research and know what you're doing before you start making pools/VDEVs because settings that you initially use might not ever be able to be changed (although this isn't particularly different from traditional RAID, other than the fact that there are less settings to set/tweak vs ZFS).

  • 9 months ago
  • 1 point

As a home server, What's all the virtualization for? Curious what all your use cases are.

  • 9 months ago
  • 2 points

Mainly containerization of all the services I want to run over a network. I.e. I have a seperate VM for my media server, network sharing software, VPN host, etc, etc. Also having a platform where I can spin up different environments on different OS is really nice as a software engineer, I can quickly test something on Windows+BSD+Linux, even going as far as to spin up a whole new VM if I need to test on a clean system.

Basically anything i want a server for, I can spin up it's own personal environment, rather than cluttering one big server running a single OS.

  • 6 months ago
  • 1 point

I'd consider upgrading your cpu. Threadrippers are nice and all, but they bottle neck that GT 710 of yours to oblivion, and I just hate seeing so much gpu performance going to waste because of a cpu bottleneck.

But in all seriousness nice NAS 11/6

  • 3 months ago
  • 1 point

did this get featured??

  • 9 months ago
  • 0 points

I really think that you spent more money on storage and you totally could have gotten 2 1080 ti in sli. But other than that, great build!

  • 9 months ago
  • 1 point

He could but that would be wasted money though. In a virtualization build as such, all he needs a GPU is for the graphics output while performance of the card itself has no matter

  • 9 months ago
  • 0 points

k, so this was a server build?

[comment deleted]
[comment deleted by staff]
  • 9 months ago
  • 1 point

Don't spam.

  • 9 months ago
  • 1 point

keep up the good work manirelli

[comment deleted]
[comment deleted]
[comment deleted by staff]
  • 9 months ago
  • 1 point

lol

[comment deleted by staff]
[comment deleted by staff]