The RX 570 PULSE ITX runs that hot even when undervolted? If so, Full Circle will most likely become a full workstation build with ECC RAM and a Pro WX5100 instead. I don't think that a RX 560 will pull off 1080p well enough to be viable.I mean the answer is pretty simple...the most powerful ITX AMD card is the R9 Nano, but it is absolutely impractical and sits somewhere between a 1060 and a normal 1070, so it's not viable for you.
The second most powerful ITX AMD card is the Sapphire Pulse 570. Decent card but runs pretty hot and as far as performance is concerned sits around the 1060...but even as an AMD/Sapphire fan I would choose the 1060 for better thermals.
The only reason I would consider an AMD GPU for an ITX build would be if I had a really nice FreeSync monitor. The only reason I would consider an AMD GPU for ATX is if I could find some 390s for cheap and crossfire them.
AMD is letting its fans (i.e. ME) down big time.
I begged Sapphire to consider making a smaller 570 Nitro under 8", because then you could at least have a really nice cooler and fans, and the 570 is a nice performance bracket. Its just their dumb pulse card runs way too hot for that budget heatsink they use. Really dumb. Keep in mind this is coming from someone who owns like six Sapphire T shirts and hats and wears them in public.
My 5c.