You're over-finessing here. Pipe smoking is niche. Philosophy, sadly, is largely relegated to being a niche discipline cloistered in what little remains of academia. Even in a campus town in Oregon, you're wanting something that probably doesn't exist.
You want a place where you can smoke your pipe and talk about books you like. That's fine. Start there. Find a book club that caters to your tastes. Build good friendships with the people there. Then maybe, just maybe, the pipe can come into play. Or maybe not. But what could happen is you invite those newfound friends to your campfire, where smoking is less of an irritant because it is done outside. Let drinks flow and snacks be plenty. Create a comfortable atmosphere around the fire, and you'll get what you wanted and more.