I don' think you are going crazy, becasue I am in the middle of doing something similar and I'm sure I got the idea from this thread at some point in the past.
I'm using this switch from adafruit. https://www.adafruit.com/product/3350
But TBH i see it as a bit of side project rather than part...