The difference is that 4 and 5 are actually decent, and 3 blows some major chunks.
edit: People are actually championing 3? Ugh, people. Just, no.
1 - Great game
2 - Modification of great game, but more a technical improvement than anything else
3 - Introduction of culture, differences in choice of civilization, and otherwise a bunch of terrible mechanics (particularly in the tech tree)
4 - Improvements on culture, more variation in unit types, unit xp, religion (that's a negative,) and a lot of little things that are an improvement rather than a detriment.
5 - Takes just about everything I ever said I wanted in a Civ game and manages to roll it together into one. Still, in spite of having the things I want all at once, it isn't inherently better.
The tech tree is reason enough to hate 3.