My experience:
I enjoy Virginia tobaccos in 9mm charcoal filter pipes. There is something in Virginia that the filters catch, that allows me to enjoy the sweetness and flavor everyone raves about. Without the filter, all I taste is harsh, acrid ash. I have three 9mm filter pipes.
I have a 6mm Savinelli Bing's Favorite that smokes sweet with the balsa filter, and horribly without, no matter what I put in it. But it is great with Virginia, VaPer blends, and Sutliff Revelation Match. I have even tried this pipe with the 6mm adapter, but it smokes better with the balsa filter.
Bitey aromatics don't bite as bad with the 9mm filter pipes.
Otherwise, I do not use a filter for Latakia based or VaPer blends, nor for some aromatics. The filters take away from those blends.
I do use the filter in my cobs, and my Dr. Grabow pipes. But honestly, I do not find that the paper filters do anything different to the smoke.
My background: I own 87 pipes, including Dunhill, high grade Petersons, Mark Tinsky and Randy Wiley pipes. I have been piping since 1976. My favorite pipes are a Peterson Grafton 999 & St. Patricks Day 150, a Neerup bent brandy, and a Rattray's Craggy Root bent pot.
As far as filters go, don't dismiss it till you have tried it. The 9mm filter is the best selling pipe in Germany and other European countries. It isn't for everyone, but it might be for you.