I recognize so much of this from my move to a smaller case earlier this year - the questions about size vs. cooling vs. noise in particular. You're "luckier" than me though: I had a bunch of hardware that I couldn't afford or didn't want to replace - among others a rather large ATX PSU and a Fury X GPU. Starting from scratch will save you a lot of hassle.
For me, silence is not negotiable. I guess I was damaged by my CrossFire HD 4850s (stock cooler) in an open-air case. My previous case was a Fractal R4, which is one of the quietest cases around, but it's ... huge, at least in my eyes, and didn't insulate out what I needed. I ended up getting an NZXT H200i (which has zero noise insulation, and even an open vent on top) and outside of wanting it to be even smaller (I suppose that'll happen when my PSU needs replacing!), I couldn't be happier. My approach ended up being what several people above have said: go over the top on cooling, make sure your components (GPU in particular) have
good coolers, slow down your fans, and that's pretty much it.
My Fury X had a whiny pump (too noisy for me; not noisy enough for an RMA), but I found an EK waterblock on clearance, so I went full custom loop. I wouldn't recommend this unless you have to/want to/have the money to burn, but exclusively having 120mm fans in my PC has made it
so damn quiet. Thanks to the fan control in the case (through NZXT CAM) I shut off the fan on the radiator after my CPU in the loop unless it's above 45C, which means at idle, I'm running two Silent Wings 3 fans at low RPM on an intake rad, cooling my entire system more than sufficiently. It's
audible under load, but never bothersome. With a 120mm + a 240mm rad, my CPU barely exceeds 60C, and my GPU never goes above 50C. While some would probably balk at my liquid temps, I feel like I've found the perfect balance of thermals and noise. I have some insulation foam panels lying around, but since assembling the build I haven't even considered bothering installing them as the PC is plenty quiet already.
I feel I've gone through the gamut of experience when it comes to handling noise: way back when, I had a Cooler Master HAF case (excellent airflow, but soooo noisy with 2008-era GPU fans! Not to mention that the air doesn't really go anywhere when your entire case is a giant vent.), a Define R4 (quiet, but that doesn't help when you have to crank up your fans and/or have noisy components), and now this. The H200i isn't open-air by any definition, and I think this helps it stay reasonably quiet, but its very open front side bezels allow for
enough airflow that two 700rpm fans keep my system nice and cool. If the case was smaller and more packed, or I didn't have water cooling, I'd probably be leaning towards more airflow and a more open design, but you don't need entirely exposed fans to not have to spin them up.
Tl;dr:
- Don't restrict your airflow excessively. H200i is fine, Enthoo Evolv ITX is likely not (I've read some horror stories). Open-air is probably even better (as long as it's not so open that all airflow dissipates immediately), but not necessary. Also, open-air requires frequent cleaning.
- 120mm fans are perfect (140mm can be noisier even at low speed due to more turbulence at the fin ends; smaller will be higher-pitched and thus more audible).
- Run your fans slow, get good fans that provide airflow at low rpms.
- Buy components with good coolers. You can't insulate away the noise from a bad cooler. If you have no choice, strap a 3rd party fan to it.
- If you're still not happy, add noise insulation. Foam+asphalt padding sheets can be bought cheaply and stuck to any case. Fractal makes some great cases, but don't let insulation restrict your choice of case.