Using dowel of appropriate diameter, it would be possible to drill it out. Depending on the wood, it might or might not be durable. In my experience, the plastic ones are not heated enough to cause any problem or impart any taste, so I'm not sure this is worth the effort, but it is possible, and somewhat easy, if you have a steady and precise hand.