There are so many factors here, it is impossible to come up with one answer. I appreciate both the idea that your home is your castle, and you should have the freedom to live your life. On the other hand, if your activities become part of the neighbor's life, then accommodations should be made.
How close are you to your neighbor's when you smoke outside? This depends on the layout of the properties. It depends on the climate, the temperature and the wind patterns. Do the breezes carry away the pipe smoke, or does it settle in the area well past your neighbor's doors, windows, and air intakes?
Are the neighbor's asthmatic or do they suffer from COPD or cardiovascular problems, or chemical sensitivities? That is, is there any real likelihood that your enjoying your pipe would actually affect some neighbor's health?
I would only ask the neighbor if you feel there is a strong likelihood that there could be a problem. Unfortunately, human nature is such that if you ask unnecessarily, the neighbor might want to be asked about other choices you make in your life -- pets, social gatherings, cookouts, etc. You want to be thoughtful but not hectored.
Like all social living, this requires good character -- both compassion and empathy, and backbone and boundaries. You have to work it out with yourself and be as generous as you can without being pushed around.