If I tried running it at 3.8Ghz would the TDP change or will it always be 65?
Would there also be any issues with stability because the jump from 3.0 to 3.8Ghz (1700) is more than from 3.6 to 3.8Ghz (1800x).
Don't quote me on this as my knowledge for overclocking is very general. In short, if you want to overclock Ryzen 7 family, go for the 1700, it doesn't matter if its base clock is lower than the 1800X, it's built off the same wafer which means it has similar tolerance. The clock difference is the result of "product binning" where AMD (or Intel), tests the processor and found at what speed/configuration the processor runs best at. For Ryzen 7, processor that runs stable at 3 GHz at 65 watt TDP goes as 1700, the one that runs stable at 3.6 GHz at 95 watt TDP goes as 1800X.
My friend who owns a computer store, an avid AMD fan and rain down on me about everything AMD. Suffice to say, all Ryzen 7 system he built overclock stable at 3.9 GHz with minimal voltage increase, even the 1700, which he built for his own daily use. So basically, the 1800X is generally not recommended if you're going to overclock as the price is too high and the benefit from the overclock is minimum. Still, there's also the golden bin or bin roulette or whatever the cool kids said it today. Some processors overclocks higher than the rest in one processor family, that one is luck of the draw.
For overclocking, you can throw TDP out the window as it will be higher than non overclocked model. TDP or Thermal Design Power is the theoretical maximum heat a processor generates when running. Matching a processor TDP with HSF/AIO rated TDP is a good practice, but you can put a higher TDP HSF/AIO to a lower processor TDP rating is still okay. Ryzen 7 processor XFR (extended frequency range) actually benefits from this since when it's needed, the XFR feature will boost the clock, basically automatic overclock, when it detects that sufficient cooling is available. I don't know if it works or not, my stock 1700X only reached 3.9 GHz on short bursts and that's with generous airflow and 120mm AIO inside my Phanteks Evolv ITX.
Still though, I'm not against overclocking, but for SFF, better just use the stock configuration. Some SFF users even underclock their system to make it as silent as possible.