I still think I remember reading that Barnabas was supposed to be around 35. Tony definitely wouldn't refer to a 25 yo as an "old man," since he was probably around that age himself.
So, we could always look at Tony's comment like this. Carolyn was supposed to be what... 18 around this time? We know there is underage drinking in Collinsport, so we have to go off of some other evidence... What was her age in the Collins Family Bible? I remember this:
[spoiler] I believe her tombstone in 1968 put her age at 18, but I can't remember. [/spoiler]
So, if Barnabas was 35, then he would be nearly twice as old as Carolyn. Thus, Tony's "old man" comment could be justified for the 35 yo age. Of course, it could be justified for an older age as well, but considering that Barnabas looks to be around 35 in 1795, I think that Tony's comment was meant in this way.