He's not excluding the Irish. He's saying that it was discrimination against a small pocket of Europeans that weren't considered white at the time. There was never a time in American history where all whites as a group (Irish, Italian, Polish, German, English, French, and so on) were discriminated against under the umbrella of "white people." There were times when some of those European groups were terrible to others, but that was more specific.
How about not being able to vote? Yes southern white men could not vote at one time. Think about it.
Because only wealthy white male landowners could vote. They weren't discriminated against because they were white, they were discriminated against cause they were broke, didn't own shit worth bragging about, or didn't have a penis.