Truth be told, Davis didn't bother me in the least on my first runthrough of the show, save his manhandling of Sabrina when he played Ned. It was later, when I started watching clips of the series again, when I really began to notice him manhandling and hairhandling.
On the subject of actors that you didn't like, though, I have more than one: Addison Powell, Sharon Smythe, both Leviathan kids, Craig Slocum, and Keith Prentice. They all grated on my nerves to no end.