When I started smoking pipes, my wife let me smoke all I wanted in the house. No complaints, no lingering smells. My wife and I took precautions to make sure the house never stunk. Not to mention the fact that I smoke Sutliff Molto Dolce which fills the air with sweet vanilla. All was well. That was until my wife's first visit to her mother's (a few months after I started).
When she came back she told me all her mother did was complain about the pipe smell on all of her clothing. She even made my wife wash her clothes multiple times while there. Call her a picky mother-in-law, but I generally like the lady. She had my wife convinced that I was contaminating the house, and, like a dog in trouble, I was sent outside.
I live in Florida where the winter weather is mild (mostly). This time of year is perfect pipe weather for me, but those summers are brutal. If I wasn't still allowed to smoke in my car, I probably wouldn't even get to come May/June.