Personally, I doubt being wireless is the issue, I always run wireless, I find people running Torrent (myself sometimes) causes issues, when I forget to shut down bit torrent. If my processor is overloaded with a video project of some sort my whole computer will lag, and if I'm hosting that will translate into perceived network lag (it's actually host computer stall)
So, rules I've learned to live by when playing serious mule. JUST PLAY MULE.

No worries about kicking observers, if things are running great, I say let them stay, but if lag is proving an issue, dump whoever is dispensable (that means observers must go)