Hopefully to provide clarity. Toxic typically means "poisonous." As far as I know, rosewood is not toxic. Some rosewoods produce allergens. Allergic reactions are not from being poisoned, they are a reaction of one's immune system to an allergen, a foreign material, like a foreign protein. Peanut allergies will produce an allergic reaction if you eat peanuts and you are allergic to peanuts. There is no poison in peanuts, but the allergic reaction can kill you from the inflammation that the allergic reaction causes. Same with some rosewood. If you inhale wood dust of some species, that can produce an allergic reaction that causes inflammation and constricts your breathing apparatus. Not good.
Rosewood furniture, as long as you don't sand it, breath in the wood dust, or eat it (goodness) can't hurt you because it has no way of getting in your system and produce a reaction. Since furniture is typically provided with a finish, you don't even really touch it. Lots of plants do make toxins, but as far as I know, the wood products from rosewood species don't have toxins. That's why we use the stuff to make things.
Having said that I wouldn't smoke a pipe of rosewood, because I don't know what burning the stuff might release. I am aware that burning poison ivy is a very bad idea. That is what I think about when I hear of rosewood pipes.