Many people don't have money to spend as if it's a leisure or sporting event. For many, their entire income goes to necessities, which they don't need to go to a mall to get, they shop at very few places. He's making the correlation between income and citizen rights, on a subsconcious level.
Same comparison could be made in another venue, saying one didn't see those people in their church or country club. The GOP perception of what being a good citizen is skewed by wealth. If you're poor, or of color, you are automatically suspect of all kinds of things that have nothing to do with reality and justify hurting those people more.
If you are dressed better, are able to afford to hang out where they can go, have more material goods, you are virtuous. These people really need to read the Founding Fathers on the equality of all people instead of the Confederate dreams.
This guy wasn't 'overtly' racist, but he is still delusional and foolish in the way he regards others. At first he seemed to say that it was nice of volunteers to bring the people food and drink. Then he started going conspiracy on the event. Even if there was a website that alerted people where to vote, it's none of his business. Ugh.