Yes you are doing this wrong. First off you do not need to store the variables server or localGamer. The entire gamerJoinedEventHandler should just be on line setting the gamer tag as a new player.
e.Gamer.Tag = new Player(e.Gamer);
You can not convert from NetworkGamer to LocalNetworkGamer if the network Gamer is not actually Local and by your description there not going to be. To find the server you just use the networkSession.IsHost during your update method. When you need to send packets to the server just use networkSession.Host. This is all explained in the Client Server Game Sample. I am not an expert but i think this is all correct.
EDIT: You should go back to your first code block and look at the client server sample instead.