10Gb/sec in ITX w GPU

[Bpeezy]

Hi all,

TL;DR how to get fast file transfers (~10Gb/sec connection) in an ITX pc with a GPU installed.

I'm new to the forum, and it has been a great resource for finding information about SFF builds. I am currently planning on breaking down my massive CaseLabs S8 workstation into an ITX gaming pc and a file server. I'm moving to a small apartment in NYC and would like to downsize a little (something like and ncase m1 and the NAS can go be tucked away somewhere). This will be my first time SFF build, and I over looked the fact that

After doing many iterations and a lot of planning I had settled on a decent plan on what to get and how to best leverage components i already have when I realized I had made a major blunder. I plan on transferring files to and from the file server fairly frequently (eg using it as the storage for the photo collection i will be editing on my main pc). I always figured that i'd just throw a pair of 10GB NICs into the computers and run a short cable....

But ITX only has one pcie slot and I won't be giving up a GPU. So far as i can tell there are no ITX mobos for intel 1151 that have 10Gbe. So the way I see it I have a few options:

1) Figure out how to set up a home made NAS to use usb 3.1 as the client and serve files over usb 3.1 to the PC. Or same with thunderbolt?

2) Use PCIe bifurcation to get both a 10Gb NIC and a GPU installed in the PC. Then I may need to choose a different case?

3) If i am already looking at a larger case, maybe switch to a mATX build. I looked around and I haven't found many mATX cases that have a very small footprint. I would really like to stay in a small elegant case like an ncase M1.


Note: The build will be using a i7 6700k (a remnant of the S8 build) so I'm trying to stay in the 1151 socket.

Thanks for reading and thank you all for the input,
Bpeezy

 

[jtd871]

The smallest OEM mATX cases I am aware of are the Silverstone SG-09/-10.
As far as non-OEM cases, the Cerberus and Project Orthrus cases are the smallest mATX cases that I know of. Both have threads in the Custom Cases section on this site.

 

[Necere]

One other option to consider: a 2 or 3 slot, compact microATX board in the M1 (which is possible due to its three expansion slots). See the thread on hardforum, and this list on the Google docs spreadsheet.

These boards only fit the M1 because of their reduced dimensions compared to regular mATX, but do note that they are pretty much exclusively designed as cut down mATX boards intended for lower budget systems, and therefore lack some of the other features of high end mATX and mini-ITX boards. That trade off may be worth it, however, if the extra expansion slot is worth more to you than those other features are.

 

[jtd871]

Thanks for bringing that up, @Necere, I had spaced on that option.

My understanding is the 10G cards require at least a x8 slot, and I'm not sure how many of the compact mATX boards offered that in the 3rd position (assuming GPU in the 1st slot closest to the CPU). If the 3rd slot is x4, there shouldn't be too much of a performance hit running the GPU there provided the end of the slot is open to accept the x16 physical card edge for the GPU.

 

[EdZ]

For PCIe 3.0, x2 is sufficient for a single 10GbE port (16Gb/s available each direction), so an x4 slot would be sufficient even for a dual-port 10GbE card. If you are not intending to use an m.2 SD (or choose a board with multiple m.2 slots), then you could use an m.2 to PCIe adapter and put the network card on a flex riser connected to that, leaving the x16 slot for the GPU.

 

[rokabeka]

:
My understanding is the 10G cards require at least a x8 slot
:

EdZ is right. Even if the m.2 adapter works in a x2 (pcie3.0) mode then it is sufficient. (some cheaper m.2 to pcie adapters used to do only x2 mode and sometimes GPUs switch back to lower speed mode, too, but I do not know the details.)

I run 2x40G out of a x16 by splitter

 

[jtd871]

I must have been thinking Gen2 speed limitations, then...

 

[Bpeezy]

Thank you all for the feedback! After looking into the things you suggested:

It looks like all the compatible mATX mobos with 3 slots only offer 1xPCIe speed on their non-16x port.
I really like the idea of using the M2 slot! But more trouble, all the m2->pcie adapters I've found are only cover one or two lanes (please link me if you know of a x4 one).
I also noticed, it seems like most of the cheaper ebay cards are on pice 2.0. At pcie 2.0 I would need 4 lanes (or 2 lanes at pcie 3.0) to fully cover a 10Gbe card, and to get a pcie 3.0 compatible card I would have to look at a more modern card (mellanox connectx-3). Most of these newer cards are more expensive and for similar price they also have 40Gbe functionality.
To me it seems like the best solution would be a x4 M2 to PCIe adapter, so please let me know if you find/know of one.

@jtd871 those are some fantastic case projects, wish i could get my hands one of em.

If after some more research if I cant figure out a good solution. I may go with plan B:
Build the ITX storage server inside of the S8 along with my workstation PC. I made the measurements and it looks like the ITX board will fit, and hopefully the single PSU can handle the two boards. This would spread out the hit on my wallet, and also allow me to wait for a good mATX project to near completion/build a custom case of my own. Money saved could help fund 40Gbe .

That caselabs S8 is a 85L monster, would really love to have something that is a little more NYC small apartment friendly.

Thanks again for all the help!!
Bpeezy

 

[Biowarejak]

Thank you all for the feedback! After looking into the things you suggested:

It looks like all the compatible mATX mobos with 3 slots only offer 1xPCIe speed on their non-16x port.
I really like the idea of using the M2 slot! But more trouble, all the m2->pcie adapters I've found are only cover one or two lanes (please link me if you know of a x4 one).
I also noticed, it seems like most of the cheaper ebay cards are on pice 2.0. At pcie 2.0 I would need 4 lanes (or 2 lanes at pcie 3.0) to fully cover a 10Gbe card, and to get a pcie 3.0 compatible card I would have to look at a more modern card (mellanox connectx-3). Most of these newer cards are more expensive and for similar price they also have 40Gbe functionality.
To me it seems like the best solution would be a x4 M2 to PCIe adapter, so please let me know if you find/know of one.

@jtd871 those are some fantastic case projects, wish i could get my hands one of em.

If after some more research if I cant figure out a good solution. I may go with plan B:
Build the ITX storage server inside of the S8 along with my workstation PC. I made the measurements and it looks like the ITX board will fit, and hopefully the single PSU can handle the two boards. This would spread out the hit on my wallet, and also allow me to wait for a good mATX project to near completion/build a custom case of my own. Money saved could help fund 40Gbe .

That caselabs S8 is a 85L monster, would really love to have something that is a little more NYC small apartment friendly.

Thanks again for all the help!!
Bpeezy

2 systems and 1 psu? Oh boy... nope on that one.

Have you considered going the virtualization route? Would still definitely allow you to connect other devices to your NAS down the line while also allowing you to downsize.

 

[jtd871]

BPlus has a M.2 to PCIe x4 adapter (with FDD power pins to power the slot) and the Molex to FDD power adapter. I believe they're listed on Amazon, but may not be in stock.

 

[CircleTect]

Is anyone else interested in solving this problem? A company I work for really needs 10Gbe + mini-itx + GPU. I'm thinking we could design and custom manufacture an M.2 to PCIe x4 adapter ourselves.

 

[jtd871]

https://www.newegg.com/Product/Product.aspx?Item=N82E16813182964 ?

 

[jØrd]

Is anyone else interested in solving this problem? A company I work for really needs 10Gbe + mini-itx + GPU. I'm thinking we could design and custom manufacture an M.2 to PCIe x4 adapter ourselves.

TBH i'd much prefer to see on board SFP+ sockets become more common on mobos that cost less than a grand and have the option of slotting in either 10Gbase-T/SX/LX/whatever as needed.

 

[CircleTect]

https://www.newegg.com/Product/Product.aspx?Item=N82E16813182964 ?

This looks fantastic, except for the tiny, horribly noisy fan. I wonder if we can mod it?

TBH i'd much prefer to see on board SFP+ sockets become more common on mobos that cost less than a grand and have the option of slotting in either 10Gbase-T/SX/LX/whatever as needed.

Why would you prefer SFP+? Adaptability?

 

[CircleTect]

The AsRock X299E-ITX/AC has 3 x M.2 slots, but doesn't have Thunderbolt or 10Gbase ethernet or SFP+. Coupling such a board with a custom M.2 to PCIE Gen 3 x4 riser, would allow you to run any x4 PCIE 3 card alongside a full GPU, such as a 10Gbe or SFP card.

Basically just repeating what @EdZ said, but this has hugely useful implications, and could help cover a lot of shortcomings in the mini-ITX form factor.

 

[smitty2k1]

This looks fantastic, except for the tiny, horribly noisy fan. I wonder if we can mod it?



Why would you prefer SFP+? Adaptability?

I have a similar mITX motherboard from Gigabyte. I replaced the fan with a 60mm Noctua PWM. Replacement works just like any other fan on a heatsink.

 

[EdZ]

For PCIe 3.0, x2 is sufficient for a single 10GbE port

Because my brain apparently wasn't working when I wrote the last post: as PCIe x2 is sufficient for 10GbE, then it should be possible to connect a card using an interface almost every ITX board has lying around unused: SATA Express. A compact version of the something like Asus Runway board would be needed, but this would provide a 'free' PCIe x2 (electrically, x4 physically) port on almost any board. Though unlike using an m.2 to PCIe adapter there are no off-the-shelf solutions I am aware of.

 

[CircleTect]

I have a similar mITX motherboard from Gigabyte. I replaced the fan with a 60mm Noctua PWM. Replacement works just like any other fan on a heatsink.

Good to know. That motherboard in question isn't ideal though. I'm looking for a solution that can add exotic i/o requirements to various mITX builds.

it should be possible to connect a card using an interface almost every ITX board has lying around unused: SATA Express.

I can't tell if this is crazy or genius @EdZ . How hard would it be to make this work with a custom daughterboard, electrically speaking? I can't imagine it would be too hard taking cues from the Asus Runway. Challenges I can see would be in making the SATA express cable assembly compact enough - it's quite a bulky connector. An advantage to an M.2 solution could be a more streamlined, compact package.

Also, from what I'm seeing with the current crop of mITX boards, having more than 1 x M.2 actually seems to be more common than the inclusion of SATA express?

 

[EdZ]

Challenges I can see would be in making the SATA express cable assembly compact enough - it's quite a bulky connector.

On the plus side, the cables are readily and cheaply available, so you just need to choose what angle you attach the connector at (there are likely plenty of right-angle and straddle-mount connectors intended for use on drive boards).

Also, from what I'm seeing with the current crop of mITX boards, having more than 1 x M.2 actually seems to be more common than the inclusion of SATA express?

You;re right, it looks like it started getting phased out around Z270 and most Z370 boards lack it (though often have secondary m.2 slots).

 

[jØrd]

Why would you prefer SFP+? Adaptability?

Adaptability and the bit where there are already existing on board implementations. No need to give up an m.2 slot (or a PCIe slot if your going to bifurcate), no need to route additional flexi riser cables, no need to fabricate a mounting plate, etc. Whilst I highly doubt it will happen my ideal solution is still SFP+ on board in consumer hardware.