Hard to tell, why you experience such behavior.
Only some of us - choosen people - are able to run at mem@3600.
If you lower the RAM clock, at SOC VID below 1.2V, is it stable then?
Edit: I see, you already tested it "everything works fine"
Could be an issue of mainboard/cpu(vega particular). It's pretty hard to guess. It could be the signal traces on the mainboard - the routing of those. They may interfere with each other on the Signal-Level, so as APU starts to have a hard time pushing all of the data around, the interferences lead to a corruption in data, which that GPU driver is able to recover from by closing Valley and jumping into desktop.
How did the overheat happen? What was exactly overheated? CPU? I mean there is only 2-3 temp sensors you can trust, rest is kinda resting at "high-level".
Maybe do 3600 at cl18?
When temperatures change, the resistance and other electric properties may change. I could imagine, the overheat degraded a VRM-vomponent for the SOC on the mainboard. You may try to measure it using voltmeter. Internal resistance, and when on - voltagedrop, etc. It has to be "debugged" with capturing some values to be able to discuss the topic properly.
There could/should be also I2C chip, which is controlling the VRM stages, which may be programmed through I2C. (Modern OC-able graphics provide that interface) so sad, I do not have a spare A300 to test the basic build.
I keep the opinion not to Over-Power the SOC at 1.2V -- this is just crazy
Imagine, stock voltage is 1.0V, which you oc, and keep auto, it's set auto to 1.1V, which is +10% more. When you have a vdrop, ist like -10% voltage, which causes instability (in my case without manual adjustment it can go down as low as 0,97x V on 1.1-auto)
By manual compensation to 1.1375V it's stable.
What you could also try (to rule out a CPU issue) -- change your RAM and CPU to other A300 based board. If it's back to stable at 3600,CL16 and 1.13125 you know the overheat targeted mainboard.
To be able to tell more about the temperatures, you have to check the Novuton chip on the mainboard. It's the Super IO chip which turns on the system and lays everything in the way to be able to turn the CPU on.
Novuton chip is also the chip, where the power buttons are connected to which is used to establish needed voltages and do low-level job in a300 to power on the platform.
This chip comes in flavors for an Intel CPU but also separate editions for AMd CPU. The CPU temp sensor is connected from AM4 socket straight to the Novuton Chip's PINs, which reads the values and reports them..
If the Novuton has even more temperature-related PINs and only one is connected, the other may return false values, since they are floating either high or low. That's why I think one value is always like 115°C. Just have no idea where it comes from.. I believe the VRMs have also a Temperatur sensor underneath the heatsink. But I think they don't get as hot as 115°C since the ops-voltagr is between 70-110°C (as an example, please check against)
So far my ideas on that issue
Hope it helps somehow