My goal is always to have the most powerful hardware in the limited space and small footprint.
So I have a Z-Cases P50 with Flex ATX Enhance 7660B 600W PSU along with a 5800x (undervolted and power limited at 85W) and a RTX 3090 at power limit of 90% and undervolted (roughly 335W power draw average).
To cool it, I have custom waterlook with Alphacool Eisbauer Extreme (Its a combo radiator/fan/reservoir/pump) with GPU and CPU water blocks, as well as a 60 mm radiator in the front. Temps for the CPU can go up to 80C, but the GPU peaks at 60C at full load. I have the Eisbauer Extreme on top of the case, and the 60 mm radiator hanging down by one of the posts with zipties. Janky setup, but works great.
Without the power limiters and undervolting, my PSU will constantly trip. I have a 4K 144hz monitor, so that is primarily why I chose the hardware above.
Honestly, like for any build you make, you have to give it some thought, and plan out your build. It may benefit from seeing what others have done to give an idea as well. Liquid cooling in SFF is probably the best cooling option as you can get rid of the huge heatsinks and fans (for example on the RTX 4090), for better cooling and smaller footprint.
If you are going aircooling, have to factor in space, airflow, noise, etc. There are always new CPU heatsink techs, such as new fan mods to have lower profile while also improving performance or keeping it the same as stock, or push/pull config on certain heatsinks. SFF community is a special one and with a bit of research you can get some of your questions answered in regards to CPU cooler performance in certain fan configurations, etc.
Power is definitely a limited factor in SFF cases. In regards to SFX power supplies, power wattage can go up to 850 (SFX)-1200W (SFX-L) and there are several new ones annouced that will supoort PCIE ATX 3.0 power cable.
If you are looking that Flex ATX PSUs, unfortunately, I don't see any options capable of running a RTX 4090. The PCIE ATX power cable is also a problem, but there are 3rd party cable and PSU makers which will be making smaller and condensed adaptors. The power cable itself lets the power hungry RTX 4000 series know how much it can safely draw and will automatically limit power based on how much is available. Also unless you "need" the new CPUs on the market currently (TDPs are significantly higher than the last generation,) I doubt any FLEX ATX PSUs can power that in addition to high range 1440p/2160p graphic card without severe undervolting and power limiting.
I will be getting a ASUS TUF RTX 4090 in the next few weeks to try and see how it will work, and maybe need to splice the cable like JayzTwoCent. de8auer also recommended to lower the power limit to 70% which can still achieve decent unnoticeable performance drop with a significantly less power draw.