My collection is slowly building, but not to the same level as Jim's. However, I do agree with him. Some of my pipes I just love to look at and I sometimes sit back and think 'what would that pipe smoke like'? But then never do.
All my pipes have been bought as estates, and it is not until I rub away the dirt that I am sometimes pleasently rewarded with a real gem.
Then comes the often very hard decision whether or not to sell it on, smoke it or put it into the 'pedestal group'.
All I can say is 'whatever makes you happy' because, for the time being, you are (as Jim puts it) the custodian, and the choice is entirely yours.
My example is a Dunhill that I very recently picked up as part of a lot of 11 estate pipes. I was overjoyed to find it, but after cleaning it up I realised that I'd never smoke it and as aesthetics go, it was not really a pipe I could see myself smoking either. So, it is now on sale and, with the money I make from it, maybe I'll pick up another gem that will be more pleasing to my eye and I'll want to smoke (A Peterson silver spigot he says hopefully).