You still run into the adaptation issue even in terms of tone.
Your ear detects sounds using hair cells in your inner-ear. These hair cells are a composed of a bundle of long haira (hence the name) and the base of the cell that detects deflection of the hairs. The hairs come in different lengths along the length of the cochlea, generally being longer at the outside/outer end and shorter and the inside/far end. The cochlea is a spiral tube filled with fluid, and this fluid is coupled to your eardrum; the fluid couples the hairs with vibrations from the eardrum. Sounds are detected by a combination of bulk fluid movement (for low frequency sounds) that shifts around a bunch of hair cells the same way, and individual hair cells of a certain length being particularly sensitive to certain frequency bands (think of it like resonance).
Why is all that important? The adaptation of your ear to different sound volumes is frequency dependant. For example: if you were to sit in a quiet room with a loud 10kHz tone playing, hoir cells sensitive to 10kHz would adapt and reduce their responsiveness (in terms of the neurons those hair cells are stimulating, the output would be inhibited and produce a slower pulse-train). But hair cells that are not sensitive to 10kHz would be mostly unaffected. When you walk out of that room, for a period of minutes, to hours (depending on what threshold you use, as the drop-off in effect is gradual and affected by the environment) your hearing will be less sensitive to sounds in the 10kHz region, but normally sensitive to other frequencies. Listening to music certainly sounds very odd!
This is important because
ambient noise has a variable spectrum dependant on contributions from multiple varying sources, so the ambient noise in one location will cause a different adaptation than the noise in another. As a simplified example: if you live near an airport, then during the day you're likely to not as sensitive to high frequencies than someone far from any flightpaths. A fan that 'whines' annoying for one person may be completely acceptable for another based purely on where they are. Then there's the room itself: the shape, dimensions and contents (e.g. lots of rough soft furnishings vs. hard flat walls) can dramatically affect the sound hear or measured, and even change it depending on where in the room you are.
Yeah, reliable and repeatable audio testing is
hard. Brains Are Weird and
your ears are fallible. That's what all the expensive sound damped vibration isolated rooms are for when testing components, and why audio engineers rely on specially outfitted listening rooms and consulting audio meters rather than just 'going by ear'.