Yeah, you'll wanna be careful. If you do use a harder tamper, what'll happen is you'll "flare" out the bowl's interior rim, they're usually cut pretty sharp where bowl meets rim.
Honestly,
a wooden golf tee works best. Or you can buy a fancy briar tamp too, lucky for me Nate included one when I got my pipe from him.
Here's a new vs year-old comparison: (RE: rim)
I've been much more dilligent about using the briar tamp since I've noticed the rim "softening"/"flaring out" - I understand it's a little harder to see with the blackened bowl in the second picture but I hope at least the first pic demonstrates how 'sharp' the edge can be when new.