my standard response is "You've apparently confused me with someone that gives a shit about what you think."
Book worthy! Added to my repertoire for when it's needed.
The last 'accusing' comment I got was from some board skaters I was kicking out of my building's driveway. I politely told them "Guys, I'm gonna have to ask you to leave, we don't allow board skating here", to which their 'cameraman' replied, "do they allow you to smoke a pipe". He even filmed me whilst I was telling them to leave; I reckon I got my YouTube 5 seconds of infamy as the oppressive smoker.
@ Seldom: I don't have more than a couple of weeks experience, but I think Germany is way different in terms of smoking that across the pond. I never got an angry glare whilst smoking in Berlin, at one point a passer-by even asked me for a light to light her cigarette. Here, you're not even allowed to smoke in restaurant patios, like you are there.