I'm acclimated to not smoking in public, indoors or outdoors. I occasionally smoke on a walk where it is fair to assume I won't be close enough to anyone to expose them to my smoke. I don't feel this as an oppressive restriction, but I can smoke at home, and I'm not especially gregarious, so that may account for my indifference.
If I go out for a meal, especially a little "fine dining," I don't expect to smoke, and I don't expect someone else to add their smoke to my cuisine and beverage menu.
Asthma, chemical sensitivities, COPD, and allergies can all be aggravated by secondhand smoke, so I can understand the sensitivity there.
I think I inhaled and ingested more nicotine as a non-smoking child and young adult than I ever have as a smoker. The sickest I ever got from nicotine was on a long bus ride from Chicago to Ft. Myers, Florida. So I have to balance that against my own interests as a moderate, one-bowl-a-day pipe smoker.