I don't think that Barnabas would have felt any compunction about killing Maggie at that point.
I don't think he was hypnotizing her in any way.
I don't think he would have had any trouble - physical or emotional - killing Sam as well.
I don't think it occurred to him that other people might come along.
I don't think he was worried about consequences. After all, he didn't think about consequences when he was planning to kill Burke in cold blood. It was left to Willie to point out the consequences to him.
I need to watch this episode again, but what I'm thinking is that maybe Barnabas was thrown off by the location. He had just come through the maze underneath the Old House that exited onto the beach. Maybe, because of that, he still had a feeling that he was in his own house - or at least in his own territory - and it was a great shock to Barnabas when Sam stumbled uninvited into that territory, to such an extent that Barnabas was slightly disoriented and couldn't think fast and clearly.