Go by taste. Different people will have different preferences for how long a filter will taste good. I would point out that most (but not all) pipes designed to use a filter will smoke perfectly well without one, so in most cases you have the option of smoking a pipe with its assigned filter, using the adaptor that can be used in place of the filter for a filterless airway, or just leave the filter out altogether and use the clear airway.
The rare pipe will have too open a draw or turbulence in the airway that you don't want, but most pipes including the 9 mm filter pipes smoke well without filter or adaptor, no turbulence, no problem. The main use for the adaptors is if you like a little more restricted draw.