Disclaimer: I am not a medical person and suggest you contact a medical doctor if you have concerns. However:
Here's what I think I know about asbestos. The particulates -- small bits that break off when asbestos is broken, sawed, drilled, mined -- once inhaled, can cause a lung cancer called mesothelioma. This is why older buildings with a asbestos insulation, flooring, ceiling panels, etc., have to have these materials removed at considerable expense. This contamination was an occupational hazard for miners of asbestos and workers in shipyards doing "ship breaking," taking apart ships constructed with a lot of asbestos, for scrap metals and other materials. Some Navy veterans who worked closely with asbestos in engine rooms and such have also had an incidence of this disease.
So an intact tobacco pipe might not present a hazard so long as it is not worn or broken in such a way as to release particulates. It is not without risk, but it may be low risk. Personally, I'd probably avoid it, but you do your own "risk assessment," as with most of life.
I have a pipe made of cocobolo wood, which can be hazardous when sawed or sanded, but mostly not when in solid form. I smoke that in a large rotation and think it is a reasonable risk. Likewise with mountain laurel, although that is a more standard material used to replace briar in the U.S. during WWII when briar was not available.