I have to admit that the first time I watched it, I didn't respond well to The Avengers.
There were things I liked about it (mostly, the dialogue and interaction) and things I didn't (it didn't seem to pull things together satisfactorily in the end; also, Loki + Chitauri why?), and then there was the biggest problem: I didn't see it until it was out on DVD, by which time I had been bombarded by .gifs and memes and all the rest. So I when I first watched it, there was some extensive fandom baggage rather blocking the view.
I still see some problems, but rewatching has shown me more of its charms. Also the fact that some things I considered plot holes could be read as character development and/or openings left for later development (hello, fanfiction). Plus, I do love the team dynamic. (Side note: If there's one thing I wish Agents of S.H.I.E.L.D. had accomplished, it's handle the "let's go
put on a show build a team!" bits with as much aplomb as The Avengers did. It seems to be getting better about this as the season has gone on.)
But I realized, too, that the things I love most about The Avengers are often the things that the movie builds on from other Marvel movies. The Thor/Loki dynamic pales when you don't have Thor backing it up (and boy howdy, does it blossom in Thor: The Dark World). Captain America's disconnectedness and the comfort with which he falls into military routine on the helicarrier (and then begins to question S.H.I.E.L.D.) mean a lot more when you've seen Captain America. I'm looking forward to Captain America: The Winter Soldier because I'm eager to see more of how Cap fits in -- and doesn't -- with the new world, as well as more of Natasha Romanoff. And of course, Iron Man gives you Tony Stark's shift from someone who doesn't give a damn as long as he can party to someone who does give a damn. (Never mind that his inability to sacrifice is rebooted for The Avengers.)
As a separate example, I've often read fanfic set in fandoms I don't know anything about. Most of the time (I SAID MOST), it's difficult to read/comprehend if you don't know anything about the original context. When you know the context and go back to read, it suddenly sparkles.
The Avengers works the same way for me: it's loads better as a movie in the context of other movies that make up its world. It's a shared-world experience. And everyone finds their own part of it that clicks.
(For the record: I haven't seen the Hulk movies; I liked Iron Man but IM2 didn't do as much for me; IM3 was in the middle and I want to rewatch; I loved Thor, Captain America, and Thor: The Dark World. Is that everyone? Good. Also I am sometimes enjoying Agents of S.H.I.E.L.D. and looking forward to the Peggy Carter TV series.)