This is really less of an issue than you think. Briar is incredibly dense and just doesn't absorb "gunk" to great degree; further, cleaning estate pipes and sanitizing them is a pretty simple process returning solid results.
Whenever the OP's question arises the folks who say, "No," to estate pipes seem to divide into to camps: Camp Creepy and Camp Gross. Since those are both personal determinations no one else should try to convince the believer otherwise. Frankly, I wish more of you would take on one of these perspectives to drive the prices of estates down, and the availability of hard to find shapes/makes up.
More than 2/3 of my pipes were estates when they came to me. All of my pipes, tobacco and shop equipment has been funded by the purchase, refurb and resale of estate pipes.
So, nope, I do not find smoking an estate pipe weird at all.
-- Pat