That's correct to a certain extent. Once the wood is out of the ground that's the end of it. How the wood is processed plays a role, the quality of the wood plays a role, the area from which it is sourced, etc, etc. But when someone refers to 50 year old briar, they're referring to the age of the burl in the ground.
Rainer Barbi wrote about what he considered the best age range for briar, between 35 and 60 years of age, and the ridiculousness of claims about 100 year old and older briar being better, because of changes in the capillary structure of the wood as it became very advanced in age that made such wood a poor choice for making pipes.
There is a certain romanticism surrounding ancient wood. In the '20's Kaywoodie claimed to use 100 year old wood. People would wax rhapsodic about Barling's old wood. But in the 1920's Barling produced a pamphlet called "The Romance Of The Barling Pipe" wherein they specified that they looked for burls that were about 60 years old. Only in their 1962 catalog do they mention wood being between 80 and 120 years of age.
There is a prime age for everything and then there is decline, whether it's wine, tobacco, wood, or people.