You should swap based on your skill and whether you think you will make a difference to the other team. You don't need to move every time teams are uneven. However if you were in a stronger team i would expect to see you move teams or ask a stronger play to move.