It starts with visuals for me - a shape that attracts my eye gets further attention. From there, I look at details and (if possible) check physical characteristics - balance, heft, etc. I'm not yet experienced enough to judge probably smoking characteristics except as they're impacted by physical stuff that I can see (off-kilter draft holes, for example), so that has to come after I've paid for the pipe...
As for what draws my eye, I started off thinking that straight billiards were boring and that bent pipes were where it's at. Now, two years in, I own more straight pipes than I do bents and I'm finding that I kind of like the billiard shape after all. I seem to prefer larger pipes - something that fills my hand, but can still be clenched relatively easily (Savinelli 320 Author shape, for example). I'll probably be picking up another pipe this weekend, as I got an unexpected bonus at work (well, the bonus wasn't entirely unexpected, but the amount was higher than last year!), and I'm currently inclined toward another Author, or possibly a bent egg/apple/billiard.