Depends on the stinger - some are screwed in, some are friction-fit, etc. I've got a Mr. Brog churchwarden that has a short stinger (friction fit), and I really haven't noticed much of a difference between smoking with the stinger and without, so I've taken to just leaving it in place. On this particular pipe, it just leaves the draft hole open instead of having a perforated bit of metal stuck in it, so the draw is slightly more open than it would be otherwise, but the difference is negligible.
On a smaller pipe, it might make more of a difference. I'll leave it to the more experienced pipers to make recommendations. If you've got a pipe with atinger, try it with and without and see which you prefer.