There's only so many defenses you can use for enslaving a whole group of people, building a whole nation on the backs of their free labor, and then treating them like second class people. Accusing them of using the race card is one.