I think it's the lack of lip syncing for me. I'm used to all my english speaking video games being either perfectly synced, or at least like not synced to any language.
The semi-synced lip flapping for anime just looks jarring with english voices most of the time.
That, and I prefer original audio for any foreign work if I can.