That would explain so many little things too. Like Banner grabbing the spear without realizing it. And Loki claiming he 'saw things' that made him go nuts. The Gem could have been possessing him
--
We are thought, and reality, and concept, and the unimaginable