Well, actually, it isn't good for him to take it either.
You really did a decent thing and it probably looks better for him in the long run that he has done this before the restraining order was issued.
I'd say you may get contacted to relinquish the firearm to the local authorities (possible). If so, take a receipt and leave it at that.
I wouldn't relinquish without the receipt and I'd give your neighbor a copy. Too often you hear of people not getting receipts for firearms transferred to the authorities and it just causes a whole huge mess.