Someone with limited social skills like Matthew would have been impressed that a high society woman would have paid any attention to him. And the fact that she had chosen him to be the caretaker/handyman at Collinwood really gave him a status that he probably had never had before. His behavior is more blind devotion that anything else. Elizabeth may or may not have used that to her advantage.
I doubt that she would have have had an affair with him, though Matthew probably longed for it.
As for a romantic interest for Elizabeth, the pickings for a woman of a certain age in Collinsport were pretty slim. Eliot Stokes seems the only logical choice. But Adriana Pena's Collinsport Chronicles offered an unusual solution. In a post 1971 story, Adam returns to Collinsport with the idea of resuming things with Carolyn. But things take an unexpected time when Adam runs off with Elizabeth.
.