I remember it being near the waterfront. It's actually not hard to find. When we needed to find it, we had just finished touring the Breakers and I simply asked one of the tour guides and she gave us easy directions. If you look for Salve Regina College (I think that's its name), you'll find it, since the college uses it I think for their music department. At least when I was there, you're free to walk the grounds around the mansion, and can step inside into the foyer.
Gerard