So ban him. Ban him like any other person who goes to games to simply cause problems. When he complains, say the safety of the players and the fans are of utmost importance. If he continues, say that if he’s concerned for his safety that it’s best if he doesn’t go.