Neighbors trees on our property?

We recently measured and staked to pull a permit to build a shed. We then learned that over the past 3 years our neighbor has planted several trees OVER the property line. When he saw the stakes from us measuring, per the twp survey, he was furious and has since refused to speak to us. (we did not ask for the trees to be removed at that time). We have since placed our shed, and he promptly removed the stakes. One month later we have now set up a badminton net, he has nightly taken it down claiming the net stakes are on his property. The stakes are next to one of his trees...which is 3 feet over on our property. Can I remove the trees, or do I need to give him notice and time to remove them himself??? Any suggestions?? He has also called the police, we show the twp survery and they leave.