Yeah but Jimmy Page bought Crowley's house and he's mentioned in Bowie's song, "Quicksand", so despite personal feelings, he's still pretty famous!
Tolkien, of course. I'm sure there are a ton of classic authors who smoked the briar, as well as many famous preachers (Spurgeon for sure). It's amazing how common the pipe was up until recent history. Been watching "Little House on the Prairie" with my daughters, and Michael Landon has a pipe in just about every episode of the first season. First season of the Andy Griffith Show had pipes on racks in the drug store and a customer asking for a pouch of tobacco.
According to the movie, "Gettysburg", General John Bufford was a pipe smoker. Of course, I doubt he smoked a Dr Grabow w/ an acrylic stem as seen in the move ;-)