Just give him some time. As you keep playing in other games and your scoring gets more balanced, those abandoned games will become a small percentage of them. If after that he keeps kicking you out of his games, just blacklist him.

And if another one is hosting and he joins, you can go throatcutter. >:3 Nya ha ha ha!