No one really cares about you smoking a pipe. It's about tobacco and society. And, interstate commerce, taxes (local and customs duties), whether or not you'll avail yourself of government funded medical moneys (Your neighbors' moneys for all intents and purposes.) should you suffer health problems, etc.
Five guys smoking pipes in front of the local Federal building is hardly gonna draw interest from the media. Well, other than humorous comments. And, the general public simply doesn't accept smoking as socially acceptable. We smokers are on the losing side of that issue. Even many of the members here acknowledge smoking is not efficacious nor defensible. All we smokers really have going for us is that it is pleasurable. Hardly a defense which can be used to combat all of the negatives attached to tobacco use.
As to various government interests in your smoking. They have a vested interested in you and your tobacco related products. You do rely on the government to see that you get your product when you order. You rely on various government agencies to insure that you are not "ripped off" when buying either mail order or in person. You rely on the government to have trade rules and treaties in place so you may purchase purchase imported tobaccos, pipes and other paraphernalia. Then there are the various agencies overseeing the raising of the leaf and production of the blends you smoke and, the working conditions of those who are in the chain. If you have a loved one close by to call emergency services (tax payer funded) should you suffer a stroke or other malady which can be attributed to smoking ... well, there is another reason for "government" to take an interest. Sure, some of the relationships cited above are tenuous but, they all form part of the basis to have an interest in your smoking and are used to justify the various, often intrusive regulations.
All life is politics and we smokers do not have the clout to stem the tide of the electorate. They have spoken, loudly and the people we elect to make the rules have heard.