If a certain person is of a minority and is behaving badly, he's going to attract neo-Nazis to his criticism. That fact doesn't mean he's not really behaving badly.
To be perfectly Logical, yes you're supposed to ignore a person's character when addressing their arguments and concerns. Politics has taught you otherwise, politics wherein it is incredibly easy to discredit someone via some nasty fact or scandal. He's a neo-Nazi, he had his doodle sucked in the ovular office, she once tried marijuana... None of these things really mean that the person's politics and policies are invalid.
But look up "Ad Hominem" and you will see some examples exactly like the denunciations you here present.