Well, their relationship was complex and as has been said, we really don't know much about the beginning of it. When Angie first appeared on the scene, Barney acted truly shocked to see her. As though he didn't know she was a servant to Josette OR the Countess DuPres. Unlikely? Yeah, unless he had never seen her in their service and she seduced him (or he her) apart from that home in Martinique. Which would explain something, I have always said that if you are madly in love with the young woman of a house, you don't seduce her maid! People here seem to think that was common, but I don't think so. It was not unusual to seduce one in your OWN household, but not when you are wooing someone else and on her turf. Oh sure, Nathan Forbes, would have. But Barney was an idealistic and romantic young man, NOT a conniving womanizer.
According to the Mars/Venus guy and my own personal experience (alas, I came of age in the seventies sexual revolution), if a man does not love you before sex takes place, he never will. There IS such a thing as being too easy and it does not inspire romance. Add to that the fact that Angie is EXTREMELY controlling and strong willed, while Barney likes his women vacuous and meek. They could have made it as a couple IF he could have pulled his head out of his own keister, and if she could have been less murderously maniacal. Since neither was quite willing to do that AT THE SAME TIME, they just didn't have a chance.