what I'm interested to know is if it's possible to have a Hot-swappable PCIe connection
No, there are server systems out there that can do hot swap PCIe, they cost more than some fairly decent cars
after shutting down and then turning back on.
This makes more sense, its not hot swap per-se but sure.
The biggest problem your going to face is that the connectors (slots & edge connectors) are not very robust and are not rated for many insertion cycles. There is a solid chance your solution would fall to bits pretty quickly. The other issue could well lie with securing the card, doing this in a way thats both quick and secure would take some careful engineering. Thinking about it more you would also need to deal w/ power, i dont remember the number of insertion cycles the PEG connector is rated at but id be willing to bet its not all that many as compared to an external connector such as USB (IIRC most USB C solutions are rated somewhere over 100k cycles where as an internal connector such as u.fl is rated around 30 usually, an m.2 slot is tends to be 60). Also, having a PCIe card that relies on PEG power in the slot but that is unpowerd at boot will cause boot errors. This isnt a problem if you intend to actually swap the cards every single time (although it could be depending on your exact implementation if your using a bifurcated riser) but that gets back to the insertion cycle issues. Its also worth noting that alot of PCIe risers arent that robust either (flexi units in particular dont tend to take abuse well) and arent designed to moved around, disconnected, reconnected, etc too often. Finally you may run into driver issues having "a Compact GPU built in" and another CPU thats only there some of the time but this would largely come down to the specific make & model of both GPU's and to an extent the OS and software stack your using.
There is also the issue of not every mITX motherboard supporting bifurcation so choose wisely, of those that do not all support it in terms of the vendor supporting you if you have problems. In computing "supported" and "sure it will probably work but your on your own if it burns your house down and eats your hamster" are sometimes very different things.
To conclude, buy a thunderbolt enclosure, nothing in the PCIe stack is designed or engineered to deal w/ this kind of use case.