I pretty much do it right after smoking, every single time, and I haven't really run into any issues with stems getting too loose or whatever. The one pipe I had with a loose stem came like that from the factory, and that's easily sorted by dampening the mortise with a drop or two of water, or putting some beeswax on the stem.
My post-smoke cleaning process starts with scraping out my bowl gently with the bowl scraper on my pipe tool to knock off any tarry lumps, then it's basically just gently removing the stem, scrubbing out the draft hole with folded over pipe cleaners until they stop coming out dirty, then I rub down the chamber with a couple of folded over pipe cleaners to get all the ash and any remaining moisture out of the bowl, and i run a last pipe cleaner through the stem/mouthpiece. That's about it, takes like 5 minutes and 3-4 pipe cleaners for the whole process.
Best time to get the gunk out of your stem is when it's still warm, moist, and willing to soak into the pipe cleaner--if you wait too long, the tar buildup gets all dry and resinous and doesn't really want to come off easy, even if you're working that pipecleaner like your pipe is the only available hooker in a liberty port.
As for how well the whole thing works, I've done exactly one deep cleaning in the last 2 years, and none of my pipes taste sour or off.