Log Cravo CG7 Rebirth: X3D, A2000, water cooling and case cramming

[mishmash]

Looks amazing. Why do you need a flow meter though? For sciense?

Pretty much. 99% of the time it is just a vanity/science thing.

Actually I want to try regulate the whole circuit on heat load. If I have flow and temperature into the rad, temperature out (which is bolted underneath out of view) then I can run the system at full and stepped loads and make some interesting control curves for the fans (in theory). I have a aquacomputer quadro which I am still fence sitting on, but that will let me also record air inlet temperature, so I have all four temperatures around the rad (the flow meter also has an onboard air temperature apparently).

 

[Fa5en]

Cool project and beautiful case

 

[BaK]

Everything looks at its place as if the case what made for this setup, clean and tidy, good job!

The PSU is directly connected to the GPU.

Did you have to raise the PSU position somehow or did it happen that this is just the perfect fit vertically?

Next step is rinsing, making cables, and the leak test.

Air leak test hopefully, without any Sopalin

 

[mishmash]

Did you have to raise the PSU position somehow or did it happen that this is just the perfect fit vertically?

Thanks!

It was the other way around. The GPU had to be tilted slightly (maybe 1mm towards the CPU?) to get the RAM cooler fitting lip to sit correctly over the top of the PSU. I'll post some more detailed pics of that arrangement next time I get to it. I think that's acceptable, I've seen much worse in the form of GPU sag elsewhere and it seems fine. The other option was to start milling and shaving the acrylic bottom plate...and no one has time for that.

I don't have an air leak tester, so I'll be hooking the CPU block pump up to an external power supply and doing it that way.

 

[mishmash]

Well - a bit of time off on holiday and right back into it. This weekend was well spent finishing the power solution and everything else.

I ordered some new power connectors...but unfortunately I must have messed up part numbers...


Due to the tight space at the front face of the case I had to run my own 24 pin cable. Was rather quick work as I did something similar on my server. The wires are 20AWG - it's enough for this system, and quite slim too.


I left the board side quite loose and not cable managed, as the tubing takes priority and the cabling needs to be able to move.
The CPU cable is routed under the motherboard and plugged in accordingly.


Here is a picture of the PSU cooling solution. I used two small adhesive pads and then a large thermal strip to ensure heat transfer:


All that was left was to put it all together! Here is the finished product, after I had bled all the air out of the system and leak tested for 20 mins or so. You can see the added Quadro controller to control the fans and also record an additional external temperature sensor outside the radiator.




I had a scare unfortunately. After powering on the system I got some small sparks and power supply clicking. Clearly I triggered the short circuit protection on the GAN HDPlex. Unplugging the GPU allowed the system to start, so I started looking here. The culprit was the shunt resistor, it was contacting the copper heat sink. A little bit of kapton tape, remount the heat sink and off we go...



It works!
So far I just got to a BIOS screen and found the minimum voltage needed to keep the DDC running.

Next steps is installing windows, overclocking, undervolting, and tuning. Might also plug the gap between the fans to minimise hot air recirculation.

Apologies for the poor quality photos, unfortunately I am rubbish at taking good shots of PCs!

 

[K888D]

Wow this is insane, can't beleive you've managed to fit all that inside the CG7, great work! Can't wait to see what temp and performance results you get.

 

[mishmash]

Windows all installed, and a few issues identified. I was having difficulty tracing why the graphics card would show anything (black screen) specifically after a reboot. Turns out it is the miniDP to HDMI cable. I tried with a cheaper passive 1080p adapter and everything seems to work, so I've ordered a new active adapter to see if that one works fine at 2K.

The set of tests below is intended to slowly work out the limits of the system. Let's be clear, there is no way this little radiator will be able to keep up with both CPU and GPU at full load, the intent is to find the radiator limit, and bring the power limits down on the CPU (PPT etc) to keep the system in control.

So with that in mind, I need to work out a series of tests to tune the system:

• OCCT Power Test
  • Intent: to check the PSU consumption (using an external power meter) and if cooling capacity is being exceeded
  • Acceptance criteria: steady state approach temperature and radiator duty
• Cinebench R23
  • Intent: to check for clock stretching when undervolting
  • Used in conjunction with core cycler etc for stability checking
• Uningine Superimposition
  • Intent: to find a good tune for the A2000 with MSI afterburner

As a starting point I've set the approach target of 10C (delta between air in and water out) with a fast fan response. The pump is stable at 75% constant rate, although I may increase this a little more as the noise seems to be the same throughout the range.

Doing a quick dummy run with OCCT Power with all BIOS settings at stock shows that the radiator limit is exceeded (as expected). The results show ever increasing approach temperatures, and failure to stabilise the GPU temperature. I aborted the test early, but I expect that we would see thermal throttling on the GPU and the system eventually reaching equilibrium in a throttled state.

The screenshot below shows my aquacomputer dashboard. It's difficult to call out a preliminary radiator capacity...I suspect the current setup can dissipate 190W, further testing will confirm the final number. The ideal is to not have a system that thermally throttles!

 

[BaK]

Do you plan on decreasing the fans speed later on?
It looks like they are at full tilt 2890RPM in your test above, which should not be easy to the ears

 

[mishmash]

Do you plan on decreasing the fans speed later on?
It looks like they are at full tilt 2890RPM in your test above, which should not be easy to the ears

That's the idea. After everything is tuned and stable the final step is to find a good approach target temperature for the fans, followed by tuning of the PID controller for the fans.

Currently at idle with an approach target of 10C the fans are at about 1200rpm, much quieter. The system is certainly not inaudible unfortunately, but I am testing it with the case 40cm from my ears on my work table, once it's in a living room cabinet and you're sitting on the couch it should be better.

 

[Smole]

Windows all installed, and a few issues identified. I was having difficulty tracing why the graphics card would show anything (black screen) specifically after a reboot. Turns out it is the miniDP to HDMI cable. I tried with a cheaper passive 1080p adapter and everything seems to work, so I've ordered a new active adapter to see if that one works fine at 2K.

The set of tests below is intended to slowly work out the limits of the system. Let's be clear, there is no way this little radiator will be able to keep up with both CPU and GPU at full load, the intent is to find the radiator limit, and bring the power limits down on the CPU (PPT etc) to keep the system in control.

So with that in mind, I need to work out a series of tests to tune the system:

• OCCT Power Test
  • Intent: to check the PSU consumption (using an external power meter) and if cooling capacity is being exceeded
  • Acceptance criteria: steady state approach temperature and radiator duty
• Cinebench R23
  • Intent: to check for clock stretching when undervolting
  • Used in conjunction with core cycler etc for stability checking
• Uningine Superimposition
  • Intent: to find a good tune for the A2000 with MSI afterburner

As a starting point I've set the approach target of 10C (delta between air in and water out) with a fast fan response. The pump is stable at 75% constant rate, although I may increase this a little more as the noise seems to be the same throughout the range.

Doing a quick dummy run with OCCT Power with all BIOS settings at stock shows that the radiator limit is exceeded (as expected). The results show ever increasing approach temperatures, and failure to stabilise the GPU temperature. I aborted the test early, but I expect that we would see thermal throttling on the GPU and the system eventually reaching equilibrium in a throttled state.

The screenshot below shows my aquacomputer dashboard. It's difficult to call out a preliminary radiator capacity...I suspect the current setup can dissipate 190W, further testing will confirm the final number. The ideal is to not have a system that thermally throttles!

This is really interesting. your methodical approach is impressive. love the build and the config. super cool to see the watercooled PSU ahah.

I'm having the same probelm as you with my card sometimes outputting black and have been wondering if it is my adapter. what cable are you using atm that works? (1080p is plenty for me)

 

[mishmash]

This is really interesting. your methodical approach is impressive. love the build and the config. super cool to see the watercooled PSU ahah.

I'm having the same probelm as you with my card sometimes outputting black and have been wondering if it is my adapter. what cable are you using atm that works? (1080p is plenty for me)

I was using a cable matters mini DP to HDMI cable (i.e. no adapter, just a direct connection). This one didn't work.

The latest one that works very well at 2k is the cable matters miniDP to HDMI adapter (this one).

Oddly the simple amazon basics adapter also worked well at 1080p!

 

[mishmash]

Well - I think I've reached the end of my testing. Time to talk numbers...

We know that this system will be constrained by two things: thermals and power. So we have a few testing and optimisation objectives:

• Overclock/undervolt the GPU as much as possible (as this is weaker than the CPU, so it makes sense to clock this up and limit the CPU)
• Check the PSU limit
• Check the radiator limit
• Limit the CPU accordingly to ensure the limits are respected under 100% CPU+GPU artificial load

At stock, the power meter shows an AC draw of 250W. This surprised me, as I thought that the 5800X3D would draw much more. This means that the PSU is well rated for the CPU and GPU combination. Thermally, it was a different story. You could see from my previous post the radiator could not keep up, even with the fans at full speed. So - we have our primary constraint - it is thermal, and we need to reduce thermals as much as possible while maximising performance.

I first looked at applying a good undervolt with PBO/CO in the BIOS, followed by PPT/TDC/EDC tuning. I collected a whole bunch of data, but I've summarised the key points below.

Test Description/Changes	Cinebench R23	CPU T	PSU T
Stock BIOS	13981	88	57.1
XMP -> Profile 1 (Enabled)	13455	86
FCLK 1800MHz -> 1600MHz	14137	87
-30 all core CO	14606	82.2
CPPC Enabled, CPPC Preferred Disabled, PPT/TDC/EDC 120/75/110	14544	81.2
100/65/90	14179	78
100/55/100	13774	67
100/55/95 (Final)	13772	68	51.3

OCCT testing with a core cycler showed no crashes at -30 all core, I guess I got lucky with this chip. You can see I had to thermally limit the CPU significantly - however the performance impact was not that much compared to stock, and I got very good thermal reduction. 5800X3D definitely runs hot! I measured the PSU temperatures with a IR gun, got a good reduction there too, but I am unsure how much of this is owed to lower loop temperatures overall.

The CPU testing revealed an issue with the way the water loop is controlled. I have set the approach target at 10C, however if the loop is below 10C and the CPU/GPU experiences a sudden load, then the fans will not ramp until the water temperature is well and truly increasing fast. At this point, the fans cannot ramp fast enough to catch it. If we set the fans to follow CPU/GPU load, then we could end up in a situation where the water is too warm after being loaded, and the fans not maintaining air flow as the processors are not ramped.

So I implemented a bit of a latching function. This will cause the fans to ramp on CPU load, however will follow the water temperature on the way down. Here is how it is set up in aquasuite playground:


Now that the fans work well enough, we can move onto GPU testing.
The first part was to run the card at stock in superposition, check the GPU T and main card voltage, and undervolt/overclock from there.
Initially I just tried to drive the full 1.05V and overclock it, however this introduced crashes and simply didn't work. For my card, I found 743mV was quite nice and solid.
I started with a mem clock of +500MHz. I experienced crashes using the core slider at the end of tests - therefore I went in and started manually setting the curve to ensure that the idle clocks were not affected. Here is the end of the results table (the start is very boring)

Core overclocking results:

GPU parameter/changes	Superposition 4K / OCCT Steady Extreme UnrealEng	GPU T
Stock	4345	70
----lots of other tests----
+345 peak, undervolt flat line at 756mV	5044	55
+360 peak, undervolt flat line at 756mV	5081	56
+375 peak, undervolt flat line at 756mV	5091 / Error OCCT	55
+390 peak, undervolt flat line at 756mV	5118 / Error OCCT	55
+405 peak, undervolt flat line at 756mV	5172 / Error OCCT	55
+420 peak, undervolt flat line at 756mV	5200 / Error OCCT	54
+435 peak, undervolt flat line at 756mV	5216 / Error OCCT	53
+450 peak, undervolt flat line at 756mV	5234 / Error OCCT

It was clear that 756mV was a little bit unstable, even if it was producing good results and no crashes.

Dropping it to 743mV at +345 peak and +500 MHz passed OCCT with a superposition score of 5074. Memory testing showed scores decreasing after +500MHz memory. Here is my final curve. You can see it gradually increases from idle to ensure idle clocks remain close to stock:


Finally we go for a full artificial load test to see if we need to limit the CPU more...looks like it's all good:


The nice thing is that there are few, if any games with cause full CPU and GPU loads - so the fact that approach is held steady at full fan speed is a good sign. Total radiator power available is 175W with this setup. Power draw from the wall is 205W in full artificial load.

Next up is cleaning up the inside slightly, setting it up as a steam box and moving it to its new home. Will try and post better pics in the next post too!

 

[BaK]

Wow, thanks for describing the whole process!

Not following everything though, will have to reread it several times!

Didn't know there was such option in Aquasuite you used to create your latching function.
Reminds me of the ETL software FME I'm using daily for GIS cartography.

With what are you adjusting the GPU values, MSI Afterburner?

 

[mishmash]

Wow, thanks for describing the whole process!

Not following everything though, will have to reread it several times!

Didn't know there was such option in Aquasuite you used to create your latching function.
Reminds me of the ETL software FME I'm using daily for GIS cartography.

With what are you adjusting the GPU values, MSI Afterburner?

I could have been more technically clear with my writing

The latching function is a bit ghetto. It basically works like this:

• A real approach (delta between water out and air in) is calculated. This will be our lagging control variable
• An artificial approach is also created. This is a leading control variable fed to the main controller:
  • The CPU and GPU core loads are imported. A 3 second filter is applied to it (basically a 3 second average)
  • The filtered load is passed through a table function. The table output only provides outputs of 60% and above. Anything below this is zero.
  • A ratio is created to basically make a small alteration to the leading approach temperature when a high load is experienced. The best way to explain this is that if our main controller has a set point of 10C, then the output of this leading variable when a load is experienced will be 11. This way the main controller thinks that the approach is too high and begin ramping the fans up
• The "latching" function is a simple "MAX" function, that looks at the real vs artificial feeding variables

How this works in reality:

1. Fan controller has a setpoint of 10C. When the system is at idle and settled the approach is actually only 8C. So the fans just operate at minimum speed.
2. I launch Crysis in ultra settings. CPU and/or GPU load spikes for longer than 3 seconds. The artificial approach is set to 11C.
3. The MAX function sees 8C from the actual and 11C from the artifical and feeds 11C to the main controller. The main controller all of a sudden sees 11C and begins to ramp the fans, even though the real approach is still only 8C (but slowly rising). Eventually the CPU and GPU calm a bit and reduce to 60% or less.
4. I game for a bit, the approach eventually stabilises at 10C (or maybe more) being controlled by fan speeds.
5. I stop gaming. The artificial approach stops being a part of the max function. If the system was heavily loaded, then the real approach will still be very high. At this point the fans look at the real approach and cool the system down even though the CPU and GPU are completely unloaded.

I think there will still be some tuning on the values with real life situations, but overall it seems to work well.

I am using MSI Afterburner for the GPU. The RTX A2000 does not have unlocked voltage control, so you need to treat the undervolting like a laptop GPU. OCCT also really pushes the GPU hard and detects errors even though benchmarks look fine. I could push the GPU harder at the expense of error generation (based on the results).

 

[BaK]

Thanks for the extra explanations!
This two approaches technique is brilliant, much more advanced than just relying on CPU and/or GPU loads.

 

[SFFMunkee]

I’m… blown away. This is a stellar build log and I’m very impressed! Great work

 

[Elaman]

I thought I was going to praise what you were able to fit in this case, but now I have to praise what you were able to fit in 205 Watts!