I really don't see any major differences between the first few seasons and the last few in terms of quality.
The TV series has stayed consistently true to the books throughout the series thus far. They aren't going into every detail of the books obviously, but they have always been able to capture the critical parts on screen quite well.
I do understand how it feels to have something that feels cult worthy that becomes mainstream. GoT was definitely one of those things... the more people that talked about it online the more people decided to give it a try and got hooked. Then everyone was in it, not just the nerds and geeks. I think this is the only difference... outside factors, but truly nothing in the show has changed to make me dislike it in the slightest.