I started smoking with no filters, then switched to 9mm charcoal (just because most of my fancy pipes were supposed to be used with a filter) and never looked back. Regarding the flavour loss, I don't feel any, while the reduction in rough edges and harsh notes is very palpable. I'm used to filters a lot. I feel that even if there's a bit more flavour in unfiltered smoke, it's lost anyway to the unmitigated excessive roughness, so I believe my filtered smokes are actually more flavourful.
I use each filter 4 times, swapping each side twice, then I discard them. Between the uses, I dry the filter off completely, keeping it for days or weeks in the drying jar. I take the filter out of the shank immediately after smoke to avoid unnecessary additional soaking.
Also, using filters helps me to clean my pipes more often. When you use a filter several times you want to have you pipe very clean from inside, or the filter would look gross already after the first smoke.