I'm pretty much like you and I've been looking for a case with the exact same specs you are.
Like
@Ceros_X said, 7L is pretty much the minimum if you want a Sentry style layout if you want an SFX/L PSU and 300mm GPU. If you go on the Zaber Sentry page you can see the component lay out tool and there's no real room to shrink. A Sentry with the dimensions of 63mm (SFX PSU height), 300mm (GPU length), and 320mm (mITX board + GPU height) would be ~6L, while the current Sentry is 6.9L. When you take into account stock cooler height and extra mm for fitting things in you get a case volume of about 6.5L, but I still haven't accounted for storage drives.
A DanCase style layout with overlapping MOBO and GPU is probably the best bet in terms of maximizing volume for all the specs you want.
https://smallformfactor.net/forum/threads/custom_mod-sfx-mini-5-92l.1786/
The CustomMOD case is 5.9L and support GPUs up to 287mm/11" (fits an EVGA 1080ti SC Black) and coolers up to 44mm but if it could accommodate the larger GPU and stock cooler it'd be 6.4L.
@sergiiua plans on adding support for a stock cooler or C7 for Rev.4 and he's pretty communicative on the thread so you may be able to get him to bump it up or make a new model with better GPU support.
https://smallformfactor.net/forum/threads/the-world-most-flexible-sff-case-louqe-ghost-s1.1781/
The Louqe case is about the same size as the Dan A4 with the same compatibilities but it can also fit a 65mm cooler like the NH L9-65, which I think is amazing at this size case. The Kickstarter was cancelled for it but they're working on a V.2 so that may be something to look out for.