There are timeline issues you're not considering here though. Although it seems like no time has passed because you're watching episode to episode and scene to scene there are weeks or months going by leaving time for travel. We've seen in the show and in the books time lapses and I think we should be thankful that they don't include them only because they may be terribly boring.