2U high-performance Ryzen 3900x server build

cromo

Efficiency Noob
Original poster
Dec 14, 2019
5
0
The story:

First of all, hello all. I have been a lurker for a while now, researching for my 2U high-performance, future-proofed, non-gaming server concept to replace my existing hardware:
  1. i7 64GB 2018 Mac Mini that I use for software engineering and music production,
  2. 32GB, 8TB WD RED, HP t730 VM server running Proxmox hypervisor, hosting pfSense, backups, and other containers.
Everything will be running under VMware ESXi, including macOS, which implied some of the hardware choices.

I have decided to go with AMX and have already purchased a bunch of components (https://pcpartpicker.com/user/cromo/saved/#view=G8PdD3 for reference).

Spec:
  • AMD 3900x.
  • Low-end GPU, no gaming at all.
    • Low TDP is preferred and realistically only AMD can be used with macOS, so most likely a cheap AMD RX 460 or 470.
    • I may want to add another Nvidia card later on for CUDA to be used with Linux
  • 1 x 1TB NVMe WD Black SN750, passed through to macOS.
  • 512 WD Blue M.2 2280. Not ideal, but got it almost for free. For the Hypervisor and other VMs.
  • 8TB WD RED I have had for a while now.
  • 64GB RAM. I got the famous Ballistix DDR4 3000 CL15 on offer.
    • I was thinking about going with 128GB, but I understand I wouldn't be able to reach high freqs and that 64GB is already a stretch on Ryzen platform.
  • ENP-7660B FlexATX power supply.
  • 2x 10GB Mellanox PCIe NIC.
My bout the three remaining parts, which are the case, the cooling and the motherboard:

The case:

I spent about a month and a lot of free time to figure out what to get. There are 2 cases that I could use and that would meet my requirements:
  1. Provide full length PCI support, or ideally mixed full/low length.
    1. I don't want to be limited to low-profile GPUs only
  2. Would facilitate a back to front airflow. The case will be sitting in a regular TV media cabinet for now and any other airflow would be too problematic.
    1. this means the fans need to be directly attached to the front plate and offer unobstructed airflow, so e.g. this Logicase is a no-go despite checking all other boxes
  3. Allow to use water-cooling if I decided it was a better option in the future for more demanding CPUs or GPUs
  4. At least 2 x 80mm fans in the front, unobstructed. Ideally 3.
  5. Flex or 1U power supply to reduce the space occupied.
I spent countless hours researching cases. Eventually I found 2 cases that check those boxes:
Chenbro RM24302
  1. A 10 y/o case, already EOL but still sold by some EU distributors. About €230, so it's fairly expensive.
  2. Has 3 x 80mm fans in the front
  3. Nice aesthetics, well made, reputable brand.
  4. Unfortunately does not allow mixed full/low height PCIs. It's either or, depending on the model ordered.
  5. Could potentially use the Slim DVD drive slot and repurpose for another full-height PCI if needed
  6. Offers hot-swap slots, but that's not really a requirement for me.
  7. Spec sheet
K245f (aka x245f)
  1. Very cheap ($70), sold by multiple OEMs, no-name.
  2. 2 x 80mm fans in the front
  3. Technically designed to support the PS2 ATX, but I am confident I could retrofit the FlexATX supply vertically directly at the back plate after cutting our some steel for the fan outlet.
  4. The side airflow entry could be used for another 80mm fan, most likely after drilling a few additional holes.
  5. Nice selection of PCI slots, very future-proof
  6. Spec sheet with dimensions

The cooling:
  • Air:
    • 3900x will work with Dynatron A24. It's designed for TDP up to 105W but is said to be capable of handling even more in an efficient front-back airflow layout that the 2U cases provide.
    • The 3 fans of the Chenbro should guarantee an ideal back to front flow
    • The 2 fans of K245f would not be in direct line with the CPU and I understand this would be a problem.
  • Water
    • Chenbro would house a 3x80mm radiator.
    • K245f would house a 2x80mm radiator + 1x80mm on the side.
      • the 80mm would be pulling/pushing the air into the case, so slightly less optimal.
    • I found at least couple options for each, Alphacool or Black Ice Nemesis offer those more known, some non-name ones are also available.
    • No idea about a pump and reservoir.
    • Potential to water-cool the GPUs as well, although not sure if that makes any sense with my low GPU requirements.
  • Noise
    • I did a lot of reading and understand that a pump adds enough noise to make most solutions louder than the equivalent CPU fan.
    • Are there pumps that are substantially quieter than other? E.g. the way Noctua is for fans.
  • OC
    • Since I'll be using it for software engineering, my use case implies a frequent CPU spinning and keeping it there for couple minutes or longer. I understand that water-cooling is better at maintaining low-temps for prolonged periods of time and helps avoiding throttling/reach high AutoOC (PBO) freqs, but would that really matter with those relatively-well aircooled cases?
The Motherboard:
  1. Full ATX
  2. Rather don't care about PCIe 4.0 for now, so x470 chipset would be enough. In light of recent AMD announcements, however, I'd rather future-proof for Zen 3, so x570 is preferred.
  3. Since I have 64GB RAM, I understand t-topology vs Daisy Chain can be an issue since I want to overclock the memory
  4. 10GB Ethernet would be nice to have
  5. Don't care about embedded WiFi, wouldn't hurt though.
  6. 2 x NVMe or even 3, if possible.
  7. Thunderbolt would be nice to have, either on-board or through the headers and additional PCIe card.
  8. High quality VRMs
  9. Stability, quality and relatively buggy-less BIOS are important.
  10. Don't care about RGB
  11. Most importantly, the motherboard should have isolated IOMMU groups for virtualization (ie. ACS support), but I understand it's a standard now: https://community.amd.com/community/gaming/blog/2017/05/25/community-update-4-lets-talk-dram

I will appreciate any input you can provide to help me with my choices. I can't wait until I finally get to build it and share my progress here.
 

TLN

Average Stuffer
Mar 9, 2020
57
32
There's AsRock X470D4U2-2T motherboard, comes with 10Gb card, supports Ryzen and got a few M.2 slots for HDD drives. It's also mATX, not sure if that's a plus for you or not. I'm pretty sure there're few ATX boards with 10Gbe onboard, and that would be my choice.
I've finished my server in SFF, but with different motherboard (and platform), but still using ESXi. You can read details here: https://smallformfactor.net/forum/t...ypical-nas-or-20tb-of-ssds-in-sff-case.12807/
I've also ran Mac OS X on Esxi in the past.

Here's few thoughts:

Spec:
  • AMD 3900x.
  • Low-end GPU, no gaming at all.
    • Low TDP is preferred and realistically only AMD can be used with macOS, so most likely a cheap AMD RX 460 or 470.
    • I may want to add another Nvidia card later on for CUDA to be used with Linux
  • 1 x 1TB NVMe WD Black SN750, passed through to macOS.
  • 512 WD Blue M.2 2280. Not ideal, but got it almost for free. For the Hypervisor and other VMs.
  • 8TB WD RED I have had for a while now.
  • 64GB RAM. I got the famous Ballistix DDR4 3000 CL15 on offer.
    • I was thinking about going with 128GB, but I understand I wouldn't be able to reach high freqs and that 64GB is already a stretch on Ryzen platform.
  • ENP-7660B FlexATX power supply.
  • 2x 10GB Mellanox PCIe NIC.


SSD: There's no need to pass through NVME drive to Mac OS. There're only a few reasons for separate SSD for hypervisor. Usually pepople install it onto flash drive, but if you planning to passthrough all USB ports to Mac OS X, SSD would be a better choice. I would recommend using different drive for hypervisor and VM datastore if possible, i.e. USB + NVME, or M.2 drive plus big NVME.
RAM: I'd go with ECC memory if possible, Ryzen unofficially supports that. It's also cheaper ($350 for 128Gb DDR4), but usually bit slower. No big deal unless you wanna play games.
Watercooling for 24/7? I'd say get an air cooler. With descent cooler performance or temp won't be an issue.

P.S. I got dozen of M.2 drives that are perfect for hypervisors. I don't have use for them and won't bother with selling 16G drives nowadays. PM me if you want one.
PPS. I'm pretty sure I got low profile videocard as well. (Radeon R5 240/HD8490).
 

cromo

Efficiency Noob
Original poster
Dec 14, 2019
5
0
There's AsRock X470D4U2-2T motherboard, comes with 10Gb card, supports Ryzen and got a few M.2 slots for HDD drives. It's also mATX, not sure if that's a plus for you or not.

I am aware of it. It's nice that it has an integrated IPMI, but I don't really need it. x470 is a minus. The price is a minus. I could get an Asrock Creator x570 for that price. The size a minus. I want to go with a Full ATX to have as versatile build as possible. The 2U cases I chose, even if relatively short (15-17") and low volume (15 and 17L) offer enough space to accommodate full-sized mbs. The 3 PCI slots are also not enough: I already need 3, leaving no room for future-proofing.


I've finished my server in SFF, but with different motherboard (and platform), but still using ESXi. You can read details here: https://smallformfactor.net/forum/t...ypical-nas-or-20tb-of-ssds-in-sff-case.12807/
I've also ran Mac OS X on Esxi in the past.

Looks amazing. While my requirements are slightly different, I do appreciate the effort you put into your research.

SSD: There's no need to pass through NVME drive to Mac OS. There're only a few reasons for separate SSD for hypervisor. Usually pepople install it onto flash drive, but if you planning to passthrough all USB ports to Mac OS X, SSD would be a better choice. I would recommend using different drive for hypervisor and VM datastore if possible, i.e. USB + NVME, or M.2 drive plus big NVME.
I actually still haven't decided about the layout and I did cut some corners explaining the storage. Yeah, I would most likely install hypervisor on a separate, small drive. I have a few small M.2 drives available, which is why I indicated that 3 x m.2/NVMe slots would be ideal. About the NVMe passthrough , that's also something I haven't decided on yet. I would normally not do it, think I read somewhere that it was recommended for macOS. Need to revisit that.

RAM: I'd go with ECC memory if possible, Ryzen unofficially supports that. It's also cheaper ($350 for 128Gb DDR4), but usually bit slower. No big deal unless you wanna play games.
Yes, am aware of Ryzen's ECC support, but I don't see the advantage of it in my case. Not running anything mission-critical and never really had any spontaneous reset or kernel panic, despite running my different servers with uptime of several months. I know it's more complicated than that, I haven't looked into it much yet, wasn't aware that ECC was actually cheaper. And I do agree about the CPU freqs. Only got these Ballistic because of good reputation and cheap price.

Watercooling for 24/7? I'd say get an air cooler. With descent cooler performance or temp won't be an issue.
I do agree and I'd go with air anytime for the simplicity, cost, reliability and low maintenance. The problem is that the Dynatron A24 is the only 2U cooler available that can handle 3900x but it gets very noisy when maxed-out. It also is not possible to install it in a front-back or back-front orientation on any motherboard... except the X470D4U2-2T you mentioned. That would complicate the airflow significantly and raise the temps. A good water-cooling would guarantee low temps and be future-proof, requiring only a change of a CPU block if I decided to upgrade the platform. Additional GPU would also not be an issue, I could daisy-chain them.

P.S. I got dozen of M.2 drives that are perfect for hypervisors. I don't have use for them and won't bother with selling 16G drives nowadays. PM me if you want one.
PPS. I'm pretty sure I got low profile videocard as well. (Radeon R5 240/HD8490).
Thanks, I think I am good on that from but I do appreciate the offer!
 

cromo

Efficiency Noob
Original poster
Dec 14, 2019
5
0
The price is a minus. I could get an Asrock Creator x570 for that price.
Actually, need to correct myself on that one. I misread. While nowhere near the Creator, it's $269 and still expensive.
 

TLN

Average Stuffer
Mar 9, 2020
57
32
Agree on pretty much everything above. If it's not mission critical and you want to keep it 2U you have only a few options: Dynatrons or watercooling.


I actually still haven't decided about the layout and I did cut some corners explaining the storage. Yeah, I would most likely install hypervisor on a separate, small drive. I have a few small M.2 drives available, which is why I indicated that 3 x m.2/NVMe slots would be ideal. About the NVMe passthrough , that's also something I haven't decided on yet. I would normally not do it, think I read somewhere that it was recommended for macOS. Need to revisit that.
I can't see any reason about Mac OS X and passthrough. But I might be missing something. My OS X was flying with regular VMDK on SSD shared with other VMs. It depends on usage, but in homelab/test environment it's quite unlikely that all VM will write to drive imho, but it depends on what you're doing.

Mind you asking, why you want to go with AMD over Intel?
-Lots of cheap CPUs (Look into QS chips for example: 12 core Xeon for $100).
-Quad channel memory. 4 channels of "slow" 2133 or 2400 should be faster then 2 channels of 2999memory.
-Better support. Arguably, I know. But google "ESXI Ryzen Pink screen". Lots of people had issues with ESXi/x470 boards, not sure if that was solved or not tho.
-Lots of motherboards available: all the features, lots of server stuff. I'm running Asus board with Dual Xeon CPUs and 6x PCIE x16 slots for example. I need to get threadripper to get that from AMD.

Personally I don't mind switching the camp, but for ESXi, non-gaming usage and Mac OS X on top, Intel sounds very reasonable.
 

cromo

Efficiency Noob
Original poster
Dec 14, 2019
5
0
Apologies for a late respone.

I can't see any reason about Mac OS X and passthrough. But I might be missing something. My OS X was flying with regular VMDK on SSD shared with other VMs. It depends on usage, but in homelab/test environment it's quite unlikely that all VM will write to drive imho, but it depends on what you're doing.
Thanks for the input, I'll definitely consider running it from a regular VMDK.

Mind you asking, why you want to go with AMD over Intel?
That is a very good question.

-Lots of cheap CPUs (Look into QS chips for example: 12 core Xeon for $100).
Wasn't aware of QS chips, but from what I am seeing not even 2 of those will beat a single 3900x in performance, at a vastly increased power consumption of ~250W vs 105,. The main reason I am upgrading from my already very potent i7 Mac Mini is to get a bump of sheer processing power, so settling for anything less would be a pointless effort. Also quietly cooling 2 x Xeon in a 2U chassis would prove much more demanding if not impossible.

-Quad channel memory. 4 channels of "slow" 2133 or 2400 should be faster then 2 channels of 2999memory.
Agreed.

-Better support. Arguably, I know. But google "ESXI Ryzen Pink screen". Lots of people had issues with ESXi/x470 boards, not sure if that was solved or not tho.
Wasn't aware of these issues. Looks like that was around the ESXi 6.5 in 2007, but I agree nonetheless about the better support. Intel also offers an unrestricted GVT-g that I could use to provide GPU to other VMs or the ability to pass the CPU directly to macOS, allowing for the nested execution. The overall better SR-IOV compatibility is also a rather popular opionion. Or the ubiquitous Thunderbolt 3.

-Lots of motherboards available: all the features, lots of server stuff. I'm running Asus board with Dual Xeon CPUs and 6x PCIE x16 slots for example. I need to get threadripper to get that from AMD.
Personally I don't mind switching the camp, but for ESXi, non-gaming usage and Mac OS X on top, Intel sounds very reasonable.
Also agreed. At the end of the day I care for the price/performance/power consumption ratio. I am ecology conscious and I figured Ryzen 9 was best bang for back right now. It's not a perfect solution, though, and I will happily switch camps in the future if needed.
 

TLN

Average Stuffer
Mar 9, 2020
57
32
Wasn't aware of QS chips, but from what I am seeing not even 2 of those will beat a single 3900x in performance, at a vastly increased power consumption of ~250W vs 105,. The main reason I am upgrading from my already very potent i7 Mac Mini is to get a bump of sheer processing power, so settling for anything less would be a pointless effort. Also quietly cooling 2 x Xeon in a 2U chassis would prove much more demanding if not impossible.
I think both Ryzen and Xeon will be faster. And you'll always want it to be even faster in the end.
Cooling Dual xeon in 1U chasis is possible, but it's gonna be loud and hot.
Speaking of performance, my dual 2683v3 gets me bit under 5000 points in cinebench R15. Ryzen 3950 gets ~3900 points stock or 4300-4400 points when overclocked. So there's no simple answer.
Do I want ryzen cpu for myself? Yeah, of course! That's my choice for workstation.
Do I need one? Probably no. Seems that existing hardware is good enough.



Wasn't aware of these issues. Looks like that was around the ESXi 6.5 in 2007, but I agree nonetheless about the better support. Intel also offers an unrestricted GVT-g that I could use to provide GPU to other VMs or the ability to pass the CPU directly to macOS, allowing for the nested execution. The overall better SR-IOV compatibility is also a rather popular opionion. Or the ubiquitous Thunderbolt 3.
I know ESXi and AMD was an issue few years back with Ryzen 1000. But it seems to be working now.
I'm also aware of huge topic here: https://forum.level1techs.com/t/asr...-server-boards-x470d4u-x470d4u2-2t/139490/935 I'm too lazy to go read 1000 posts there since I have hardware already, but I'd recommend reading last 10 pages to see if everything is working right.
With Intel setup on the other hand I have all the stuff available, even though I don't need it: SR-IOV, GPU passthrough, nested virtualization, etc.


Also agreed. At the end of the day I care for the price/performance/power consumption ratio. I am ecology conscious and I figured Ryzen 9 was best bang for back right now. It's not a perfect solution, though, and I will happily switch camps in the future if needed.
For workstation I'd go with Ryzen, hands down. For virtualization at home to run Mac OS X I'd take Intel. But that's my $0.02.
It's hard to beat old Xeon price-wise: 12 Core 4650 for $100. 64Gb of RAM(quad channel) for $160. Motherboard for another $150 or so.
 

cromo

Efficiency Noob
Original poster
Dec 14, 2019
5
0
Cooling Dual xeon in 1U chasis is possible, but it's gonna be loud and hot.
Speaking of performance, my dual 2683v3 gets me bit under 5000 points in cinebench R15. Ryzen 3950 gets ~3900 points stock or 4300-4400 points when overclocked. So there's no simple answer.
For workstation I'd go with Ryzen, hands down. For virtualization at home to run Mac OS X I'd take Intel. But that's my $0.02.

Yeah, I researched the E5s after your first post and the numbers make absolute sense - although more so for 2683v3 than for 4650v2. If it wasn't for the fact I had already purchased 3900x (for a low price of $420 w/tax), I would reconsider. Especially the additional quad-channel, cheap RAM would be nice. My original idea was to put the 2u case and eventually into a rack with other audio equipment, so it needed to run as cool and silent as possible, but now I realize that even the 2x120W TDP (vs Ryzen's 105W) wouldn't be much of an issue for my planned water cooling setup, considering I don't care about the GPU performance.

I'm too lazy to go read 1000 posts there since I have hardware already, but I'd recommend reading last 10 pages to see if everything is working right.
Thanks, I'll take a look. The fact that both Torvalds and Kroah-Hartman just switched to Threadripper workstations should speak to the stability of the platform, though, so I am not afraid about this. Worst case scenario I switch to Proxmox from ESXi and deal with slightly bigger macOS maintenance overhead.
 

TLN

Average Stuffer
Mar 9, 2020
57
32
Yeah, I researched the E5s after your first post and the numbers make absolute sense - although more so for 2683v3 than for 4650v2. If it wasn't for the fact I had already purchased 3900x (for a low price of $420 w/tax), I would reconsider. Especially the additional quad-channel, cheap RAM would be nice. My original idea was to put the 2u case and eventually into a rack with other audio equipment, so it needed to run as cool and silent as possible, but now I realize that even the 2x120W TDP (vs Ryzen's 105W) wouldn't be much of an issue for my planned water cooling setup, considering I don't care about the GPU performance.
I meant 4650v3. It's 12-core CPU compared to 14 Cores in 2683v3. Clocks are pretty close, but 4650v3 is way cheaper. May be because 46xx are meant for quad-cpu servers and not many people aware that it will work in regular PCs.
If you already have 3900x it makes no sense though. It looks like a great buy, and it will be faster then those Intels for sure!

Thanks, I'll take a look. The fact that both Torvalds and Kroah-Hartman just switched to Threadripper workstations should speak to the stability of the platform, though, so I am not afraid about this. Worst case scenario I switch to Proxmox from ESXi and deal with slightly bigger macOS maintenance overhead.
Those guys run linux on bare metal. ESXi is a bit picky.
 

aquelito

King of Cable Management
Piccolo PC
Feb 16, 2016
952
1,122
If you are not afraid to mod the case, you could use a standard 240 or 360 radiator placed horizontally to cool you hardware.
Any PWM pump can be run to be inaudible. I run my dual DDC pumps at 30% : absolutely silent.

Also, the ENP-7660B FlexATX is pretty noisy IMHO. The 40mm fan makes a kind of rattling sound, even in idle.
I would use a SFX or SFX-L PSU (again, this may imply some modding).