Because when they watch sports, they see a disproportionately high number of black people. They also know that black people were forcibly bred to be beasts of burden "back then" or believe that Africa is a harsher environment, physically speaking, than Europe or Asia. So they're under the impression that black people were generally bred, through artificial or natural selection, to be more physically fit.
So, they associate those ideas.
I mean, you can say it's incorrect, but you shouldn't be surprised that it is still a thing people would believe.