Log (Completed) Shallow 2U: myElectronics high density server build

[mishmash]

Hello all,

The time has come to chase more storage density and a little more volume, as well as start consolidating as many services as possible into one server box. I've started to outgrow my Cravo CG7, so I started planning a quiet 2U server that can go in my small rack in my entertainment cabinet. Unfortunately, there were no good contenders when I searched. I could only find either the mini 202b or the IPC-G225. Then one day, as I started another work day, I saw this case from myElectronics finally pop up: the short depth 6870.

I ordered one immediately on release, and it came shipped a couple of days later. Unboxing is fairly uninspired, which is fine, we are dealing with a rack server box after all, nothing complex here:


The case is gorgeous. At the front, we have a central vandal black power button, with a fascia that is made of the grill and also keystone holes (keystone mounts are behind the fascia. This means that any keystones will be nice and flush. The two vents are for 80mm suction fans.


At the rear we see an 80mm exhaust, 2x LP slots, a motherboard opening, 6x keystone openings, and room for an SFX PSU. It would have been interesting to see a version with a PCIe utility mount instead of the keystones, however keystones offer some nice flexibility when it comes to running HDMI and USB from front to back etc.


The rough plan is to increase my storage drive number by x2, and also add 5x 1G NICs + 2x 10G NICs which will allow me to shut down my multigig switch (I'm moving to OpenVSwitch as the number of wired devices in my increased electrical cost apartment is minimal).

Still waiting on a few things in the mail, but next post will be some interesting details on the case.

 

[BaK]

a quiet 2U server

An SFX PSU should indeed be quieter than the usual PSUs for servers!
And that's not the three 80mm (Noctua) fans that should increase the noise so much!
Looking forward to seeing what hardware you are going to put in this case!

The time has come to chase more storage density and a little more volume, as well as start consolidating as many services as possible into one server box.

Can you tell us more about what this box is going to deal with?

 

[Factor Jay]

Looking good!

 

[mishmash]

An SFX PSU should indeed be quieter than the usual PSUs for servers!
And that's not the three 80mm (Noctua) fans that should increase the noise so much!
Looking forward to seeing what hardware you are going to put in this case!


Can you tell us more about what this box is going to deal with?

Sure - obviously this will be running 24/7 in a living room environment, so silence is key.

The server itself will be running proxmox - a virtualisation software. The VMs I run are basically:

• OPNsense - firewall with direct ISP fibre connection. This means I manage my own optical termination and do not have a ISP box in my setup
• Pihole - self explanatory
• Unraid - this is my NAS, and also runs my docker services. The usual suspects such as plex, grafana, *arr dockers, downloaders etc. One of my more used dockers strangely is a recipe manager! It also functions as a constant time machine backup for the macs in the house.

The current server internals will be transplanted - it is a 10th gen intel i5-10600T which most people choose for the quicksync transcoding capacity - this means the iGPU is passed through to the unraid VM which can be used in docker.

My current homelab stack draws 67W idle, and consists of the following:

• Server + NICs
• external class C+ ONT (hopefully can come back to a SFP class B+ ONT)
• PoE+ 10G/multigig switch (to be eliminated)
• Wifi 6 AP
• ISP TV box

I have no need for multigig switching, as I have no physical devices higher than 1G. Further to this, the switch idle power is a fair amount. The challenge I have set myself is to try reduce this idle power value by eliminating the multi-gig switch. As a result, some of my devices will have to either go on wifi (which is fine for me, they are non critical), or they will be run using a virtual switch via proxmox + opnsense. The wifi AP can be supplied with a PoE+ injector.

Other current hardware specs:

• 64GB RAM
• 4x 1.92TB samsung enterprise SSDs (another 3-5x to be added)
• 4x intel i350 NIC
• 2x samsung evo 970 NVMe SSDs running in zfs RAID 1

 

[BaK]

Merci pour la réponse détaillée!

OPNsense - firewall with direct ISP fibre connection. This means I manage my own optical termination and do not have a ISP box in my setup

Sounds good not to need the ISP box!
I guess you have a daughter card that lets you attach the fiber connection to?

One of my more used dockers strangely is a recipe manager!

Along with the sauces, the Bocuse and other cook books in your shell, looks like you are not only into computers but also a gastronome!

The current server internals will be transplanted - it is a 10th gen intel i5-10600T which most people choose for the quicksync transcoding capacity - this means the iGPU is passed through to the unraid VM which can be used in docker.

Good to know this CPU is performing well for transcoding!

My current homelab stack draws 67W idle, and consists of the following:

Already quite a decent idle value, curious to see how low you will manage to go without the multi-gig switch!


I'm also in the process of installing Proxmox and an OPNsense VM but into a more modest hardware (quadcore Celeron / 256GB mSata / 8GB RAM).
A freePBX VM for my VOIP phones will also be added, but right now I'm a bit struggling to choose between all the extra network stuff, like piHole, Adguard, Unbound, Bound, DNScrypt, geoIP, etc. I still need some readings to figures this all out before choosing the right tools!

 

[mishmash]

The front of the 2U enclosure is a false face, it can be removed, and we can see how myelectronics has installed standoffs to keep the keystones nice and flush:


Front removed:


A lot of long lead items came in over the past week, so I can begin measuring and fitting. As you may have worked out from the amount of I/O listed, I'll need to use a bifurcation riser in this build. So the first step is sorting out the second PCIe slot which will be running at x8 off the riser. As the server is currently running, I have disassembled my little gaming PC to use the ITX board as a fitting tool...

The standoffs are female to female, and attached using countersunk screws from the bottom. An ADT x8 to x16 (mechanical) riser is shown installed:



Tight turns, but looks like it works ok:

Sounds good not to need the ISP box!
I guess you have a daughter card that lets you attach the fiber connection to?

The fibre is a GPON connection, if you know the way your ISP authenticates itself onto the upstream fibre tree (OLT), it is possible to use an open ONT (the same device that is soldered inside the ISP box) to pass the same authentication parameters to the upstream OLT. Then it's a case of using OPNsense to perform the ISP authentication itself (i.e. PPPoE or DHCP). These ONT devices can either be external (i.e. fibre in, ethernet out) or SFP profile, which looks like an optical adapter that you can insert into a specific 10G card which allows changed 10G parameters. This allows you to take advantage of WAN speeds higher than 1G if your subscription allows it. Of course, all of this can easily be avoided if your ISP offer bridge mode on their box! (which mine sadly does not).

Along with the sauces, the Bocuse and other cook books in your shell, looks like you are not only into computers but also a gastronome!

I am indeed - although only an enthusiast, not a chef!

I'm also in the process of installing Proxmox and an OPNsense VM but into a more modest hardware (quadcore Celeron / 256GB mSata / 8GB RAM).
A freePBX VM for my VOIP phones will also be added, but right now I'm a bit struggling to choose between all the extra network stuff, like piHole, Adguard, Unbound, Bound, DNScrypt, geoIP, etc. I still need some readings to figures this all out before choosing the right tools!

I used to run a similar fanless i5 PC. They are quite cool devices, and both OPNsense and pfsense are quite able to run on a celeron, provided you have no need for deep packet inspection (which not many people do). OPNsense will run adguard, unbound and I think also DNScrypt, so it should work well for you!

 

[mishmash]

Next step is making some aluminium brackets for the power supply and overhead PCIe x4 NIC.

For the PSU, I have gone with HDPlex's new GAN 250W unit. I used some 10mm wide alu strip and bent it into brackets, and drilled and tapped holes as needed. The bracket was painted with some high temperature spray. I also hacked up a piece of scrap alu and painted it to make room for the AC power entry:


It will sit to the side of the case:


Here we can see how the fans will interact with the PSU as well as space for the first set of 5x SSDs:




I will use 4x PCIe lanes to run a 4x 1G intel NIC. This will be internally mounted, and then patch to keystones run to the back panel. The card will be mounted overhead the motherboard, for this I made another bracket. My original plan to use a 5cm riser did not work as it would not flex back enough, but luckily I had a spare 3M riser which fits perfectly. The riser is mounted under the bracket with standoffs, and the bracket itself has tapped bolt holes to allow direct mounting to the case.



Here we can see another small metal bracket running off the PCIe blanking plate. The purpose for this one is to provide a support to the overhead PCIe bracket and hold it in place. If I ever need to replace this card in the future, the standardised mounting is an advantage, however there is a small reduction in card length (approx 25mm) due to the need of setting it backwards. In any case, most 4x1G cards are smaller than standard LP sizes.



With all the metal work done, I can move to creating my custom power cables. Here is a picture with all 4x fans fitted, I put a spare SSD in the second drive bay. A nice feature of this case design is that the PCIe cards are also cooled with the left-front fan, this will allow me to change the cooling solution on the 10G card to be installed later.

 

[BaK]

The fibre is a GPON connection, if you know the way your ISP authenticates itself onto the upstream fibre tree (OLT), it is possible to use an open ONT (the same device that is soldered inside the ISP box) to pass the same authentication parameters to the upstream OLT. Then it's a case of using OPNsense to perform the ISP authentication itself (i.e. PPPoE or DHCP). These ONT devices can either be external (i.e. fibre in, ethernet out) or SFP profile, which looks like an optical adapter that you can insert into a specific 10G card which allows changed 10G parameters. This allows you to take advantage of WAN speeds higher than 1G if your subscription allows it. Of course, all of this can easily be avoided if your ISP offer bridge mode on their box! (which mine sadly does not).

Thanks for the explanation! No clue how my ISP does the authentication though... Well my subscription is 200Mbps only anyway.
My first thought was no ISP box means not having to deal with their hotline in case of problem, but now I see you are doing this for the extra speed!

My ISP box does actually allow bridge mode, but for that I had to call them to put the box back to IPV4 since the bridge mode was not available in IPV6.
Actually it is still in router mode... because of my kid's Nintendo Switch! While it is possible to go online (Nintendo eShop, online account, etc.) with my actual Netgear router, the latter doesn't let it go through for online gaming, other players never show up. So as this major crisis had to be solved urgently, I've chosen to attach the Nintendo to the ISP box directly.
I will see if I can sort this out when my OPNsense router will be up and running in place of the Netgear one.
But after all, having the ISP box as a router with mine right behind it does not seem to be problematic and should even add security. Or am I missing something?

Next step is making some aluminium brackets for the power supply and overhead PCIe x4 NIC.

For the PSU, I have gone with HDPlex's new GAN 250W unit.

Very nice, thanks for all the pics! Very clean job with all the custom brackets and good to see this new PSU in action!

 

[mishmash]

My ISP box does actually allow bridge mode, but for that I had to call them to put the box back to IPV4 since the bridge mode was not available in IPV6.
Actually it is still in router mode... because of my kid's Nintendo Switch! While it is possible to go online (Nintendo eShop, online account, etc.) with my actual Netgear router, the latter doesn't let it go through for online gaming, other players never show up. So as this major crisis had to be solved urgently, I've chosen to attach the Nintendo to the ISP box directly.
I will see if I can sort this out when my OPNsense router will be up and running in place of the Netgear one.
But after all, having the ISP box as a router with mine right behind it does not seem to be problematic and should even add security. Or am I missing something?

I think you should be able to do everything through OPNsense and just have the ISP box in bridge and doing nothing else.

From my experience with my xbox, there's a few things to look at, and will also probably need some research:

• DNS provider - check to make sure that your device is happy with your chosen DNS, or that pihole isn't blocking something it shouldn't be. I added certain xbox addresses from the pihole whitelist page - I can't find the link right now, but I recall it was a github page of some sort. Plenty of pihole forum posts on different devices also for whitelisting.
• Firewalling - perhaps the most important one. Xbox has a list of default ports it needs forwarded to run its xbox services. Also, certain games require certain ports also. Typically you can find these on the port forward site or reddit/forums. I would recommend not opening 80 and 443...these are not required, even if it says so (or only required as outbound).

It could be possible your netgear was blocking something. Most ISP boxes come configured for UPnP - which would let the nintendo device through. Maybe your netgear didn't have that enabled? OPNsense does come with a UPnP plugin too, I don't like using it as it's a little less secure than strictly port forwarding, and I could never achieve "Open NAT" status with it on my xbox. It could be a good solution for you though, as you can just set UPnP on specific devices/IP addresses with OPNsense.

Sounds like your issue was probably the second point. I bet there is a port that needs to be open for multiplayer...

 

[BaK]

I think you should be able to do everything through OPNsense and just have the ISP box in bridge and doing nothing else.

Okay, I'm going to aim at that!

Sounds like your issue was probably the second point. I bet there is a port that needs to be open for multiplayer...

I will keep the port forwarding or UPnP in mind once the Netgear router will be replaced by OPNsense. Should indeed be the issue as I don't have a piHole running yet.
Thanks a lot for having looked into it!

 

[mishmash]

Back after a bit of a break over Christmas.

Against my better judgement I decided to move all system components to the new case last night. Before I left for holiday, I had measured and built some nice PSU cables. Unfortunately I miscalculated the quantity of super slim cable, and had to use silicone cables for the sata and CPU EPS cables. This was actually a blessing in disguise, as the sata cables are super flexible and you can make ladder connectors out of them nicely. All the cables were designed to be tucked away nicely to avoid airflow issues.


Here is the case almost fully assembled. The 10G card is slightly bent...I think my second PCIe slot is sitting at an angle, so I will need to look at this and remedy it. The 4x1G card is not yet installed in this picture.


I managed to get the system online late last night, with some tweaks to the BIOS (bifurcation) and IOMMU groups (had to append the less secure PCIe ACS override patch). Good news is proxmox can see and split the two network cards with no issues! Bad news is either the M2 sata card I have is DOA or it is not receiving power or something. I have a spare NVMe SSD I will use to test the bifurcation slot and see if it shows up.

Next post - hopefully solutions and final pictures!

 

[confusis]

This is f'n amazing. Love a good rackmount build!

 

[mishmash]

Small update - I ended up finally moving my whole arrangement to a "switchless" stack. This means I run everything from one box - router, NAS, services etc all converged.
Disadvantages: everything goes down when I work on hardware.
Advantages: for a small apartment a switch isn't needed, particularly when I run a very beefy wifi access point which handles vlans. The only device that is wired is the xbox.

Total idle for ONT, decoder, server and wifi AP is 58W. Could be a bit lower, but the 11x SSDs do add up. I will have to do the cost benefit of a lower quantity of higher capacity drives vs what I have now.

I racked it all up neatly in my home entertainment cabinet:


At this point the hardware side is done! I can play with adding more services etc this summer, and next winter might be some more hardware updates/changes. Thanks for following all!

 

[confusis]

58W for such a unit is very impressive. Well worth it!