After the events in Sinnoh with Cyrus (who they took credit for defeating, technically not a lie since they were involved) they headed back to Kanto and it was there they were given some advanced training. Then they arrived in Unova where Ash just happened to be headed and they actually were competent. Didn't get blasted off til like episode 9 or later. They only reverted to their normal uniform after the earthquake resulted in the anime team cancelling the original arc with team plasma. That's how we ended up with the meloetta and Giovanni arc where we saw Giovanni's Persian is absurdly powerful. Like, there's been E4 members who didn't do as much harm to Pikachu with their ace Pokemon as Persian did.
Then for Kalos and alola, team rocket also just so happened to be assigned there. None of that "following Ash" business. Actually, even for hoenn they were heading there to help expand ground for team rocket to set up and operate there. So all things considered, because of Ash and Co stopping evil organizations and teams, the team rocket organization has been able to (implied, quite successfully) move in to those regions and set up operation.
So yeah, team rocket actually isn't as incompetent as we think they are. Plot armor for Ash though. :/
persian literally only pushed pikachu back against a tree and then giovanni put them in a box ---