I think that ashy flavor you're getting comes from either leaving the flame over your tobacco too long, or not giving the tobacco time to cool between relights. However, the first step to getting past that bad flavor and the need to relight in general is paying more attention to your packing and other initial prep work.
Like others have suggested, you should try packing the pipe a little looser than you think with less tobacco than you think as well. I would also suggest you pay more attention to your false light. Some folks don't think it matters, but a proper false light has always helped me get an even light and a cool smoke with minimal relights as well. I typically make sure that the entire top layer of tobacco has been fully scorched. I then give it several steady puffs, some light tamps to keep it leveled, then I let the pipe sit and cool for about 10 min. If I'm smoking in my truck, I go through this process in the house before I hop in. When I go back for the true light, I circle the flame 2 or 3 times making sure I don't leave the flame over the tobacco for too long; it's almost like I'm just tapping the tobacco with the flame while taking is strong, sturdy, and steady draws. I rarely have to relight after that. If I do relight, I let the pipe cool a bit before I do. And I won't dump my ash until I'm at least half way done. Like one gentleman said, the ash is your friend. Just a super light tamp after the relight and we're off and smoking again.
With a little practice and patience, soon you'll be able to sit your pipe down for nearly five minutes and come back to it; it'll still be smoking with a little coaxing from your tamper, or two of you fingers covering the bowl while you give it a few sturdy draws. This is just what works for me; it ain't gospel.