Provided you aren't purposefully aiming your cameras into their home or enclosed backyard you are fine.
As to the camera that goes into their front yard, you really should have it as the secondary or "over spill" and have the camera focused on more of your property. Thats just down right rude the way you describe it.
Also, I OC quite a bit. Neighbors, even rude ones and I had them in Hammond, don't respect people, but they do respect the gun. Sad but it's the truth.
Problem here is, since you probably haven't been OC'ing from the start, it's going to have the same effects as if it was from the first day they saw you.
I digress, keep you cameras focuses on your house and adjoining common areas and you'll have no worries.