I don't think we ever see Barnabas in something as casual as a sweater. Except for David, we don't see any of the Collins men in anything less formal than the aforementioned dressing gown or smoking jacket. I think we see Roger in pajamas and robe during the haunting of Collinwood but as I recall the pajamas seemed very well pressed.