Paratroopa: Opposite for me actually. Rock Band had three things going for it in the challenge department - new instruments to learn, stricter timing windows, and DLC that came out pretty fast that was harder than almost anything GH3 offered. Challenge isn't really the reason I dislike GH3.
New instruments: true, but apples to oranges. If you want to compare the two, then it's about the guitar. Besides, vocals aren't anything you "learn," and bass is just an easy version of guitar in RB, so that really just meant drums.
Stricter timing windows: I can't argue that. Making the transition is difficult, however...
Hard DLC: There may have been some seriously hard DLC, but the base games were pretty pathetic. Not counting GG&HT and maybe F/LT and RttH, I was completely disappointed with the challenge factor on all the songs on guitar. I sightread all of them except for GG&HT, and while I'm extremely good, I'm not a complete natural. By contrast, the on-disc songs for GH3 were much more challenging, albeit not nearly as bad as GH2, where Carry on My Wayward Son was in the second tier of eight. (At least it was in the PS2 version.)
Now, where GH3 was a total fail was on the DLC that was available. I actually bought my 360 when GH3 and RB were coming out specifically because I wanted to have access to DLC. Before that I was content playing everything on my PS2. And, well, GH3 had complete crap in the DLC department. The few songs they did have were typically ok, but there was almost none of it. It wasn't until World Tour that they started getting any kind of library, and it paled in comparison to Rock Band's by that point.
I think that in terms of being a guitar-playing game, GH3 was vastly superior to RB. The instrument was way better. (I don't understand how anyone plays on those damn stratocasters!) The menu system was cleaner, the graphics were more aesthetically pleasing, and it had a progression that actually felt like I was doing something besides playing In Bloom, Maps, I Think I'm Paranoid, and Say it Ain't So over and over and over and over and...and that's assuming you could find someone to play with since you couldn't do the tour mode by yourself. (I eventually bought a mic stand.)
In terms of being an anything-else game, GH3 was vastly inferior to RB. It wasn't a great party game. It didn't have drums or vocals. RB's DLC catalog eventually came to be the overreaching decision factor between the two. But GH3 itself was pretty good.