Like a few other forum members have mentioned of their own pipe practices, I rarely remove my stems from my pipes. I probably take a stem off any given pipe once every couple months. If you use a pipe cleaner or two after each smoke, as you should, then I just don't see any reason to remove the stem very often at all, and especially not while the pipe is still warm.
For those of us who remove stems only on occasion, the question of whether it is safe to remove a stem from a warm pipe is a non-issue.
For those of us who remove stems only on occasion, the question of whether it is safe to remove a stem from a warm pipe is a non-issue.