Or the implication here is that most of white America and specifically Moderate America is racist, or at least only really cares about their own (white) race. I don't think that necessarily follows. I think a lot of those in the middle might well be offended if he had made it specifically about black and Hispanic people. Also - there are Hispanics and Blacks in the middle and upper class so you would have to trim them out.
As for his own base, well that would have made it more palatable to them; but they are largely fine with the comment as it is.