I've been grappling with this lately. I've only been smoking a pipe regularly for 2 years, with an occasional cigar thrown in. I used to be am occasional/social cigarette smoker but quit that in 2006. I had a bad case or bronchitis 6 weeks ago which hasn't fully resolved, and have had nasal congestion the past week and had a bad bout of that last year as well. I normally rarely if ever get sick.
Everyone is different. For me, even without inhaling I find that pipe smoke gets into my throat and lungs and inevitably in nasal passage when I retrohale. And I do find it irritates me if I smoke with any frequency. So I am committing to only smoke a pipe only very occasionally when I get over this persistent cold. For me that means once, maybe twice a week.
But I really do believe that everyone's body is different. I enjoy pipe smoking and it sounds like you do as well. As someone said, it seems virtually impossible that any harm will result from short term smoking. So try it out, see how you feel and how your body reacts, and adjust accordingly while being aware of the risks.